COMMENT ⊗   VALID 00010 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.require "memo.pub[let,jmc]" source
C00003 00003	.bb INTRODUCTION
C00012 00004	.bb WHY ASCRIBE MENTAL QUALITIES?
C00022 00005	.bb TWO METHODS OF DEFINITION AND THEIR APPLICATION TO MENTAL QUALITIES
C00050 00006	.bb EXAMPLES OF SYSTEMS WITH MENTAL QUALITIES
C00083 00007	.bb |"GLOSSARY" OF MENTAL QUALITIES|
C00101 00008	.bb OTHER VIEWS ABOUT MIND
C00102 00009	.PORTION NOTES
C00111 00010	.bb REFERENCES
C00112 ENDMK
C⊗;
.require "memo.pub[let,jmc]" source;
.once center
draft
.skip 20
.cb ASCRIBING MENTAL QUALITIES TO MACHINES

Abstract: Ascribing mental qualities like ⊗beliefs, ⊗intentions and
⊗wants to machines is correct and useful if done conservatively.
We propose some new definitional tools for this: second order
structural definitions and definitions relative to an approximate
theory.
.skip 5
(this draft of MENTAL[S76,JMC]@SU-AI compiled at {TIME} on {DATE})
.skip to column 1
.bb INTRODUCTION

	To ascribe certain %2beliefs%1, %2knowledge%1, %2free will%1,
%2intentions%1,  %2consciousness%1,  %2abilities%1  or %2wants%1 to a
machine or computer program is %3legitimate%1  when such an ascription
expresses  the  same  information  about  the machine that it expresses
about a person.     It  is  %3useful%1  when  the  ascription  helps  us
understand the structure of the machine, its past or future behavior,
or how to repair or improve it.  It is perhaps never %3logically
required%1 even for humans, but a practical theory of the behavior of
machines or humans may require mental qualities or qualities isomorphic to
them.  Theories of belief, knowledge and wanting can be constructed for
machines in a simpler setting than for humans and later applied to humans.
Ascription of mental qualities is %3most straightforward%1 for machines of
known structure such as thermostats and computer operating systems, but is
%3most useful%1 when applied to entities whose structure is very
incompletely known.

	The above  views are motivated by work in  artificial intelligence$
(abbreviated  AI).  They can  be taken as asserting that
many of the philosophical problems  of mind take a practical form  as
soon  as one  takes  seriously  the idea  of  making machines  behave
intelligently.  In particular, AI raises for machines two issues that
have heretofore been considered only in connection with people.

	First, in designing intelligent programs  and looking at them
from  the outside  we need  to determine  the conditions  under which
specific mental and volitional terms  are applicable.  We can  exemplify
these  problems by  asking  when might it be legitimate  to  say about  a
machine, %2" It knows I want a reservation to Boston, and it can give
it to me, but it won't"%1.

~Work in artificial intelligence is still far from showing
how to reach human-level intellectual performance.
Our approach to the AI problem involves identifying the intellectual
mechanisms required for problem solving and describing them precisely.
Therefore we are at the end of the philosophical
spectrum that requires everything to be formalized in mathematical logic.
It  is sometimes  said  that  one
studies philosophy  in order to advance beyond  one's untutored naive
world-view, but unfortunately for  artificial intelligence, no-one has
yet been able to give a description of even a naive world-view,
complete and precise enough to allow a knowledge-seeking program to
be constructed in accordance with its tenets.
~
	Second, when we  want a %3generally intelligent%1$ computer
program, we must build into it a %3general view%1 of what the world
is like with  especial attention to facts  about how the  information
required to solve problems is to be  obtained and used.  Thus we must
provide   it   with   some   kind   of   %2metaphysics%1 (general
world-view)  and %2epistemology%1 (theory of knowledge) however naive.

~Present AI programs operate in limited domains, e.g. play particular
games, prove theorems in a particular logical system, or understand
natural language sentences covering a particular subject matter and
with other semantic restrictions.  General intelligence will require
general models of situations changing in time, actors with goals
and strategies for achieving them, and knowledge about how information
can be obtained.~

	As  much  as  possible,  we  will  ascribe  mental  qualities
separately from  each other instead of bundling  them  in a
concept of mind.   This is necessary,  because present machines  have
rather   varied  little   minds;  the   mental  qualities   that  can
legitimately  be ascribed to them are few  and differ from machine to
machine.   We  will  not  even try  to  meet  objections  like,
%2"Unless it also  does X, it is illegitimate to  speak of its having
mental qualities."%1

	Machines  as  simple  as thermostats  can  be  said  to  have
beliefs,  and having  beliefs seems  to be  a characteristic  of most
machines  capable of  problem  solving  performance.    However,  the
machines mankind has so far found  it useful to construct rarely have
beliefs  about beliefs.    (Beliefs about  beliefs will be  needed by
computer programs to reason about what  knowledge they
lack and where to  get it).  Mental qualities  peculiar to human-like
motivational structures, such  as love and hate, will not be required
for intelligent behavior, but we could probably program  computers to
exhibit them if we wanted to,  because our common sense notions about
them  translate  readily into  certain  program and  data structures.
Still other mental qualities, e.g.  humor and appreciation of beauty,
seem  much harder  to  model.   While  we will  be  quite liberal  in
ascribing ⊗some mental qualities  even to rather primitive  machines,
we will  try to  be conservative  in our  criteria for ascribing  any
⊗particular qualip	d\~∀~(∪)QJ↓gkGG∃ggSm∀AgKGQS←]f↓←LAi!SfAa¬aKdA]SYXA≥SmJAAQSY←M←aQS
CX~∃¬]HAβ$AeKCM←]fA→←dACMGeSE%]NAE∃YSKMLAi↑A5CGQS9KfXAQo↑A]∃nAM←I[fA←_~∃IK→S]Si%←\Ai!ChAg∃KZA]∃GKgg¬erAM=dAIK→S]S]≤A[K]QCXAcUCYSi%KfAC9H~∃KaC[aY∃fA←L↓iQKSHAkgJ0AKqC5aYKf↓←LAgegiK[LAi↑A]QSGP↓[K]i¬X~∃cUCYSi%KfACIJACg
eSEK⊂XAg←5JAMSIghACQiK[aQfACh↓IKMS9S]NA∧AmCe%Kir~)←LA[∃]iCX↓ckCY%iSKf0Ag←[∀AGeSQSGSg5fA←L↓←iQKHAmSK]fA←\↓[K]i¬X~∃cUCYSi%KfXA9←iKf0AC]H↓eKMKIK]GKL\~∀~(]'↔∪@A)≡A
∨→+≠8@b~∀9∨≥π
↓π≥)∃$~∀_]ED↓/⊃2A¬'π%∪	
A≠9)β_AE+β→∪Q∪&}4∀~∀~(∩Jg/!r@Ag!←kYH↓oJAo¬]hAi<@ACg
eSEJ↓EKYS∃MfAi<@A[C
QS]KL@ACh↓CYX}∀b~∃)!SfASLAiQJ↓←aa←MSiJAEkKgi%←\Ai<AiQCPA←L~(JeeK⊃kGiS=]SgZ∀b\@A%]giK¬HA←L↓CgWS9NAQ←\A[K]QCXAcUCYSi%KfAG¬\AEJ4∀Jge∃IkGK⊂JbAi<AaQsMSGCX↓←]Kf0AoJA¬gVAQ=nAi↑JgCg
eSEJ∀bA[K9iCXAEkCYSQSKf~)i↑Aa!sgSG¬XAgsMiK[f8@A)Q%fAck∃giS←8A[Cr↓EJA[=eJAgQeCSO!iM←e]CeH~)C]HA5CrAY∃CHAi<AEKiQKdAC9goKeLAi↑AQQJAcUKgiS=]fA←_AeKIUGiS←9SgZ\4∀~∀∪Q↑AakP@AiQ∀ASggUJ@Ag!CeaYdXAG←9gSIKH@ABA
←[akQKdAaI←OeC4@AM←H~∃oQ%GP@A]JAa←MgKgf↓G←[a1KiJA1SgiS9Of\@A)QJ↓EKQCYS←dA=L@Ai!JAae=OeCZ↓S\~∃¬]rAK9mSe←9[K]h↓SfAI∃iKe[%]KHA→e←Z@↓iQJAMiekGQkeJA=LAiQ∀Aae←≥eCZ@↓C]H~)GC\A	J@AM=k]HA=kh@A	rAgS5kYCi%]N@AQQJAC
iS←\↓←L@AQQJAaI←OeC4@AC]⊂AiQJ4∃K]m%e←][∃]h@A]SiQ←Uh@AQ¬mS]NAi↑A⊃KCX@↓oSiPAC]rAG←]
Kah@↓←LAE∃YSKL8~∃≥KYKeiQ∃YKgf0AiQKIJ@ACIJ@Ag∃mKeC0AeKCM←]f@↓M←d@↓CgGe%ES]N↓EKYS∃L@AC9H~∃←QQKdA5K]iC0AckC1SiSKLt~∀~(]∪)5>`v~(∩F\A¬YiQ←UOPAo∀A[Cr↓W]←n↓iQJAAe←Oe¬ZXASQfAgi¬iJACPABAO%mK\A5←[K]P~∃Sf↓kgkC1YrA]=hAISIKGiYdA←Eg∃emCE1JXAC9HAiQ∀AMCGQfAoJ↓GC\A=EiCS8~∃CE=khASQfAGkIeK]h↓giCi∀A[Cr↓EJA[=eJAe∃CISYdAKqaIKggK⊂AErA¬gGeS	S]N~)GKei¬S\AE∃YSKMLA←dA]C]if↓iQC\↓S\AC9rA←i!KdAo¬r\~∀4∀∩F\↓mK\↓SLAo∀AGC\↓gS[k1CiJAQQJAS9iKeC
iS←\↓←LA←UdAae=OeCZ4∃oSi ASif↓K]mSI←][K9hAkg%]NAC9←iQKHA[←e∀AG←[AeKQK9gSmJ↓ae←OICZXAQQJ~∃MS[kY¬iS←\↓[CrA	JABA	SYYS=\AiS5KfAi=↑AgY=n\@A]JACYM↑~∃[¬rA]←PAQCm∀AiQJ↓S]Si%CXAG=]ISi%←]fA=LAiQ∀AK]m%e←][∃]hA←HAiQJ↓K]mSI←][K9hOf~)YCof↓←L~∃5←iS←8AS\@↓BAgk%iCEY∀@AM←IZXAo!KeKCL@ASh↓[Cr@↓EJAM∃CgSE1J@Ai<A[CW∀@AB~)aeKI%GiS←8A←L@↓iQJA∃MMKGQfA←L↓iQJ@↓EKYS∃MfAo∀ACgGISEJAQ↑@Ai!JAae=OeCZ4∃oSi!←khA¬]rAG=[aki∃dACh↓CYX\4∀~∀∩\Aβg
eSES9NAEK1SKMf↓[CrA¬YY←n↓IKeSYS]NA≥K]Ke¬XAgi¬iK[K9if~∃¬E←kh↓iQJAAe←Oe¬ZOfA	KQCm%←dAi!ChAG=kYHA9←hAE∀A←Ei¬S]KH↓Me←Z↓C]r~)MS]SQJA]k5EKdA=LAgS5kYCi%←]f\4∀~∀∩\@@AQQJAE∃YSKL↓C]HA≥←CXAMiekGQkeKf↓oJACMGeSE∀Ai↑AQQJAaI←OeC4~∃[CdAEJA∃CgSKHAi↑AU]IKeMiC]H↓iQC\↓iQJA⊃KiCS1fA←LAae←≥eCZA¬fAKqAeKgg∃H~∃S8ASif↓YSgi%]N\~(~∀∩F8A)QJ↓EKYS∃LAC]⊂AO←C0AgieUGike∀ASfA1SWKYdAi↑A	JAGY=gJAi<AiQJ4∃gieUGike∀AiQJ↓IKgS≥]Kd@↓←LAi!J@AaI←OeC4AQCH↓S\@A5S]HX↓C]H@↓ShA[¬r@AE∀~∃KCMSKdAQ↑AIK	kNAi!JAae=OeCZ↓S\Ai∃e[f@↓←LAi!SfAgQekGiUeJAi!C\AI%eKGi1r~∃MI←Z@AQQJAY%giS]≤\@@A%\@AM¬GhXA%h@ASLA←Mi∃\@Aa=ggSE1JAM←H@Ag←5K←]J↓i↑~∃
←eeK
hABA→CkYh↓ErAe∃Cg←]%]N@A%\AOK9KeCX↓iKe[LACE←UhAiQ∀@AS]→←e[CQS←\~)S\AB↓ae←OICZA←HA[CG!S]JX↓ISCO9←gS]≤@AoQ¬hASf↓oe←]≤ACfA∧AMCYMJAEK1SKLX4∃C]H@AY←=WS]NACh@↓iQJ@↓IKiC%Yf@A=L@@AQQJ@AAe←Oe¬Z@A←H@A[C
QS]JA←]Yd~∃gk→MSGS∃]iYr↓i↑AI∃iKe[%]JAQ=n@Ai!JAMC1gJAE∃YSKL↓SfAe∃aeKg∃]iKHAC]H4∃oQCPA[KG!C]Sg4AGCkMKHASPAi↑A¬eSgJ8H~∀~(∪βYX↓iQJA¬E←mJ↓eKCg=]fAM=dACg
eSES9NAEK1SKMf↓CeJA∃aSgi∃[←Y←≥SGCX8~∃R]∀\ACg
eSES9NAEK1SKMf↓SfA]∃KIKH↓i↑AC⊃CahAQ↑AYS5SiCi%←]fA=\A←kH~∃CE%YSir↓i↑AC
ckSe∀AW]←]YKIO∀XAkg∀AShA→←dAaIKISGQS←\X↓C]HA∃giCE1SgP~)OK]KICYSu¬iS←]LAS\AQKe[f↓←LAi!JAKY∃[K]i¬erAgQekGiUeJA←_AiQJ↓ae←OICZ\~)!KeQ¬afAi!SfASLAiQJ↓OK]KICXAe∃Cg←\↓M←dA¬gGeS	S]NA!SOQKHAYKm∃YfA←_~∃←e≥C]Su¬iS←\↓i↑AgegiK[L\~∀~(4∪)Q%f@AW%]H@A=L@Ai∃YK←Y=OSGC0@AC]¬YsgSL@ASfA←Mi∃\@AkMKMkXAS\~)k]IKIgiC]⊃S]NA9Cike¬XA←e≥C]Sg5fACf↓oKYX↓CfA[¬GQS]∃f\@A!KeJ@↓Km←YUiS←\4∃iCW∃f@Ai!JAaY¬GJ@A=L@AI∃gSO\↓C]H@↓oJ@A=MiK\↓k]IKIgiC]⊂@AiQ∀@AMk9GiS←8~∃aKIM←e[∃HAErAC\A=eOC\↓EKM←IJAoJAk]I∃egiC9HASiLAIKi¬SYKH↓aQsg%←Y←Od\~∃)∃YK←Y=OSGC0AC]C1sgSfASf@↓CaaY%GCEY∀Ai↑@↓agsG!←Y←O%GCX@↓C]H@↓g←GS¬X~∃a!K]←[∃]BAS8@Ag↑↓MCdA¬f@Ai!KgJA¬eJAI∃gSO]∃H@A←HAQCm∀AEKK8@Agk	UKGh↓i↑~∃MKYKGQS←\\4∀~∀∪!←oKm∃dX@AQKYK←1←OSG¬XAC]¬YsgSLAMCS1fAoQ∃\ACaAYSKH↓i↑ACMaKGiL~∃←L↓]CikIJAoQ%GP@A!CmJA9KSiQ∃dAEK∃\@AI∃gSO]∃HA]←HAae←⊃kGKH↓Er@A9Cike¬X~∃g∃YKGi%←\AMI←ZAB↓a←ak1CiS←8\@@A5kGPA5KISKYCXAg
SK]G∀AoCf↓ECgK⊂A←\AQQJ~∃)kIK↑5πQeSMiSC\5≠←gY∃Z@AQea←iQ∃gSfAQQCh@↓iQJA⊃KiCS1fA←LAiQJ4∃o←e1HAoKIJAIKMSO]K⊂AErA≥←H@A→←dAi!JAEK9KMSh↓←LA[¬\\@@↓)QJAMie←]≤AM←e4~∃←LAiQSLAQsa=iQKg%fAoCLACEC9I←]K⊂@ACh↓iQJAQS[JA=L@A∂¬YSYK<AC]H↓≥Koi=\~∃EUhA←G
CgS←9CYYr↓eKGkIf\@@↓¬CeedAπ←[5←]Kd≥f@PbdndRA¬qS←Z↓←L@A∃G←Y←≥r~∀E9Cike∀AW]←]fAEKMhD@AMKK[f↓i↑AE∀A[SgQCWK]1rAECMKHA←8@AiQ∀A]←i%←\Ai!Ch~∃9Cike∀@ACf↓B@Ao!←YJ@↓SfAi!J@Ae∃gkYh↓←L@A¬\AKm=YkiS=]CerAae←
KgfAQQCh~)gKYK
iKHAQQJ@E	KghA9Cike∀D\4~(~∀∪π=[aki∃efAO%mJAe%gJAi<A]k[∃e←kf↓KqC[AYKfA=LAEk%YIS]≤ABAQ%OQKd4∃gieUGike∀A←\AQQJAE¬gSfA=LABA1←oKd↓C]HA
←]Ik
iS]N↓gkEg∃ckK]PAC]C1sgKf4∃kgS9NAiQ∀AQSO!KdAgQekGiUeJ\@↓)QJA≥K←[KQerA←_AiQJ↓KYKGQeSFA→SKYILAS\~)BAie¬]gSgQ←dAC9HASiLAGQK5SGCX↓G←[a=gSiS=\AOSYJAeSMJAi↑↓SifAAe←aKIiSKf4∃CfA¬\AKY∃GieSAGSe
kShA∃YK[K9h\~∃QeC]g%gi←eLACeJ↓G←[E%]KHA%\Ag[¬YXAG%eGkSQfAC]⊂Aa←o∃eKHA%\Agi¬]IC`⊗ 4+←∂KMβSzβ7π/*β3?∨N≠π1β.c↔7↔w#MβO.≠!βπ~αε:∪~aα>K~aα:>'→βπ;"β≠3'αk≠3?π→84*≡{7CW&+KMβ∂∪∃β∪/≠'∨;.!β←'&AβS#/≠∃β3};'∂πbβ↔3↔n+;SLhSS=β}∪↔eβ
β∪↔OO∪↔⊃β␈∪∪↔Iε≠?∪∃ZβS#∃ε#↔O'>s↔Iβ/≠Wπ3gIβ;↔.#9∨PhS∂?;≡K∪↔Iπ##∃βπ∪?C↔↔#'↔Mε{→βSF)βSK∞sO'O&{KMβ∂→β∂'⊗≠W'Qε+3↔7.sSM8hRS#∃ε#↔O'>s↔Iβ}1β¬βFK∨#↔∩β3↔[.aβ3πv;Wπ∨*β←?K←→β←'&AβS#*β?K∪/⊂4+∂}#∃βπv!β∪?/≠9∨QεCπ[∃π#=β/v{]βπ⊗{WQβ&C∃αεt#Mβπv!α>K≠YβS#*βWO↔∩β?_4W##∃βFK∨#↔∩β?K∪/⊃β3πv;Wπ∨*β;↔↔&q∨Qβ↑s?]β&C∃β∂}kCWS/⊃∨Mβ␈∪∪↔Iε≠?∪∃ph(4(LK9βSF)βπ␈3∃β∂∂≠↔M1π+O↔K~β?→β&C∃↓βFK∨#↔∩β3↔[.aβ∂πrβ∂?7εc↔S↔gH4+'>s?K∃αβS#∃εc?←↔∩β3↔[.a1β.≠πWO*βS#∃αβ↔#∂3'?Iε{→βSF)β#'>C↔I↓εc↔[↔bβOgO&+5β'_h+∂?oβ3↔S.ceβ∪/#↔K7Ns↔⊃β↔I↓βSF)β[πg+↔Mβ}1↓βSF)β#'>C↔Iβf+[↔1αβ[πKN3↔≠X4+∃v99↓↓εK9β?⊗#↔Iβ&yβ∪↔&+K7'v)↓βSF)β?W&≠?7∃ε{→β¬αβ∂?7π+S↔IπβK?∨⊗51β}s∀4+v+↔∪9?!β∂?w≠'∪↔∩βS#∃ε33'An33?C~q↓α#␈;↔[↔∩aβ←#.p4+←*βπO∂⊗K∃βn+;SπbβOSK.≠SWK*βS=βG+7π;~β?Iβ>{π3Mπ#=↓β≡{∂'↔'I1β←*βπ3←∂KL4+>+Qβ#N;#3eαβ';∂}kC3↔&)βOg∨#↔7MZβS#∃αβ#'∨F+Iβ3/3↔1β⊗+#π[N{I↓β≡;;?"β∀4V3W33JβCK↔&K∂S↔"↓β≠K}i↓β#N;#↔Iαβ3↔[.aβ?≡+K[π&K?;Mαβπ;⊃αβ#'∨F+I↓βf+[↔0hQ3π?→	β↔6+9β←F+9βSF)βW;&+K3gNs≥β3␈;↔Iβf+[↔1ε∪↔#π6K?IβO→β∪↔&+K7'vS∃8hP4(&⊗+O'∪/→βS#*βπ?6)βC#Nc?O?εC'∂πbβK↔π≡{;Mβ6{Iβπ≡≠K'Ns≥β7.sSπ0hSGWπfKS'↔~βS=βn∂#'v+M1αJβO#πfaβπK?+∃βSFQβ'rβ?K∪/⊃βS=εkπ/∃εkπ∂#Ns↔L4V∪↔#π6)β';&+33'>+;S3Jaβ←∃π;'31εCπ[∃π#=βCgram them to ascribe beliefs
etc. to each other and to people.

→→→→Here there will be more on machine's models of each others minds.←←←←


]'↔% A)≡↓π∨→+5≤@b~(_]ED↓)/≡A5)⊃∨⊃&A∨↓	
∪9∪)∪∨8Aβ≥λ↓)⊃∪HAβ!!1∪πβ)%∨≤A)<A≠≥Qβ_A#Uβ→∪)%&~∀4∀∪∪\↓←kdA=aS]S=\XAB↓[CU←HAg←kIGJA←_Aae←	YK[f↓S\AI∃MS]S9NA[K9iCX~)C]HA=iQKd↓aQSY=g←aQ%GCXA
←]GKAifASLAiQJ↓oKCW9KgfA=LAiQ∀A[Ki!←IfA=L~∃I∃MS]SQS←\AQQChA!CmJA	KK\@∀eKqa1SGSi1rJbAUgKH\A)QKIKM←e∀AoJA%]ie←⊃kGJAQo↑~∃9KnHA-S]If↓←LAI∃MS]SQS←\tJegK
←]HA=eIKd↓giek
ikeC0AIKM%]SiS=\JbA¬]H~∀∀eIKM%]SiS=\AeK1CiSm∀Ai↑A¬\ACaAe←qS5CiJAQQK←edJbAC9HAiQ∃SdACAaYSG¬iS←\4∃i↑A⊃KMS]%]NA[∃]iCX↓ckCY%iSKf8~∀5≥=mKYidASfA9←hAC	g←YkQKYrA≥kCeC9iKKH84~∀~(]∪)5>`v~(~∀F\Jg'K
←]HA=eIKd↓'iek
ikeC0A	KM%]SiS=\\Jb4∀~∀∪MiekGQkeCX↓IKMS9SiS←9fA←L↓ckCY%iSKf↓CeJA≥SmK\↓S\Ai∃e[fA=LAiQ∀~∃gi¬iJA←_AiQJAgsgQKZAE∃S]NA⊃KgGe%EKHA]QSYJ↓EKQCYS←eC0AIKM%]SiS=]fACIJ~∃O%mK\A%\AiKI[fA←_ASif↓CGik¬XA←d↓a←iK9iSCX↓EKQCYS←dH8~∀~∀$5¬KQ¬mS←e¬X@AI∃MS]SQS←]f↓CeJA=MiK\AMCm=eKHA%\@Aa!SY←g=aQr\Aα~∃MsgiK4ASf@↓IKMS9KHAi<@AQCYJ@AB↓GKei¬S\@AEkCYSQrASLAShA	KQCm∃f@AS8@AB~)GKei¬S\Ao¬r@A←H@ASf@JeI%ga←g∃HJbAQ↑AEK!CmJ@↓S\ABAGKeQCS\A]Cr\~))QKSH@A←gQK]gS	YJ@AYSeik∀@ASfAG←]MKemCQSgZv@AiQ∃r@AI=\Oh@↓a←giUYCiJ4∃S]i∃e]CXAgiCQKfAi!Ch@A¬eJAk9←EgKImCEY∀@Ai↑AaeKMK]hAMGSK]
J@AC9HA[Cd~∃eK5CS\AU]←Eg∃emCE1J\~∃!←oKm∃dXAgUGP@A⊃KMS]%iS←]LACeJ↓CoWo¬eHAM=d@A[∃]iCX↓ckCY%iSKf0AEKG¬kgJX4∃CfA
←[[←8AgK]MJ@AgUOOKgQfXABA[K]QCXAcUCYSid@A[Cd@A]←P@AeKMkYh@↓S\~∃	KQCm%←dXA	KGCkMJAC]=iQKd↓[K]i¬X@AcUCYSidA[Cr↓aeKm∃]hASPvAJ]≤\@A∩↓[Cr~)iQS],As←j↓CeJ@↓iQSG,[QKC⊃KHXA	khAa=YSiK9KgfA5CrAaIKmK]PA[rAMCsS]≤Ag↑\4∃!CeQSGkY¬d@AI%MMSGUYiSKL@AGC8@AEJA←mKIG←[J0@AEkP@AC\AS[aIKggS=\@A←_~∃mC≥kK]KMfAeK5CS]f8@A)Q∀AYSW%]N@A→←dAE∃QCmS=eCXA⊃KMS]%iS←]LAgiK5fAMe=Z~∃G¬kiS←8XAEkP@A∩A]←kYH↓S]iKIaeKh↓gGSK9iSMSAKqa∃eSK]
JACf↓gQ←o%]NAi!Ch~∃	←YI]∃gfAS8Aa←gQkYCi%]N@A
←[aY∃pAgiIkGikIKfA←_Ak]←	gKem∃H@AK9iSiS∃f@Z~)ae←m%IKHA%hASfACGG=[aC]%KHAEdABAo%YYS]≥]Kgf↓i↑@AQCWJA	CGVA5SgiC-Kf@Z4∃SfA5←eJA1SWKYdAi↑A	JAeK]CeIK⊂AErAU]IKeMiC]I%]NA←_@AC]⊂@AG←9ie←XA←mKH~∃]CQkeJ@AiQC8@@ASL@@Aa=gSiSYSgiS@@Ai%[SISQr\@@A∪h@↓Sf@AACeiS
kYCe1r~∃S9giek
iSmJ↓i↑AS5COS]∀ABAI∃iKe[%]KHA	KQCm%←eSgPAies%]NAi<AMSOUeJ@A=kh~∃¬\@AK1KGie=]SF@↓G←[aUiKd\A)es%]NAi<AIKM%]JAK¬GPAcUCYSidAEKQ¬mS←e¬YYr~)o←kY⊂AOKh↓QSZA9←oQKIJv@A=]Yr@↓gS[k1iC]K=kgYrAa←gQkYCi%]N@A∧@AG←5aYKp4∃gieUGike∀@AS]
YkIS9N@A[∃[←er0ACeSQQ[Ki%FAk]%hXAG=]ie←0AgieUGike∀XAC]⊂~∃S]Akh[←Uiakh↓o←kY⊂AsSK1HAae∃ISGi%←]f@↓iQChAG←k1H@AE∀@AG←5aCeK⊂@AoSQP~∃KaaKeS5K]h\4∀~∀∪QQKeJ↓SfAB↓gK]g∀AS\A]QSGP↓←aKe¬iS←]¬XAIK→S]Si%←]fA¬eJA]=hAiC-K\~∃MKeS←UgYrA∃mK\A	rAiQ∃SdAaI←a←g∃ef\@↓'kaa=gJAg=[K←]∀AOSm∃f~∃C8A←aKICiS←9CXAI∃MS]SQS←\A=LAYK9OiP@!J]N\↓S]m←1mS]N↓BAGKIiCS\↓aYCi%]kZA	CdRX4∃C]H↓BAoQ=YJAg
Q←←X↓←LAa!sgSG%gifA¬]HAa!SY←g=aQKeL~∃EK
←[Kf↓ckSi∀ACii¬GQKH↓i↑ASP\@Aα↓MKnAeKCef↓YCiKHXAg←5K←]J↓KYgJ↓GeSi%GSuKL~∃iQ∀AIKM%]SiS=\ACf↓YCGW%]NAg=[JAI∃gSeC	YJAaI←aKeQrXAaI←a←g∃fABA
QC]O∀X~∃C9HAiQ∀AGQC9OJASLACGG∃aiKH8@A)Q%fASf↓]←e[¬XXAEUh~∃S_AiQJ↓←eSO%]CXA⊃KMS]%iS←\↓Kqae∃ggKH↓oQCh↓iQKr↓eKCY1rA[K¬]hAEd~∃iQ∀AYK]≥iPXAQQKrA]←kYH↓eKMkMJAi↑↓GQC]≥JXACIOkS]≤AiQCPAiQJ↓]KnA
←]GKAh~∃[¬rAQCYJASiLAkgKLXAEkPAShA%g\Oh↓oQCh↓iQKr↓[KC\↓Er@E1K]Oi D\@AQQSfAMQ←of4∃iQCPAiQJ↓G←]G∃ahA←_@EYK9OiPD↓CfAB↓ae←a∃eirA=LA←E)KGif↓SfA[=eJAgQCEYJ4∃iQC8AC]r↓←aKe¬iS←]¬XAIK→S]Si%←\\~(~∀∪π¬e]C`αβ#πMε9β'w#↔K↔∨#';≥π≠↔∂SN{9β'r↓∃J7.;';:βπ;⊃∧s↔∂↔∨≠'Se+λ4+↔w#'S3.!↓
SF)α∂?v≠↔CQε{→α'w#↔;ON{9β≠␈⊃β¬α⊗{?Q∩β'9β>C'∂!εC∀4+n/↔Mε	βO'nK3πIπβ?';"βOπgNs≥1↓+⊃
'QεKMβ∂f+πIβ&CπQβ&C∃β7/##?⊂hS?→β∨#KW∂'+Kπ1ε;π3O≠'M1εK→βππβ3'∂∞∪3∃1εKMβ7␈∪∃βC␈;↔K≠.aβS#∞p4+SF)β↔F['?⊗KOS'~β7↔SF{⊃1β⊗+∂πW≡)β'Qε≠π9β∨+CC3Jβ¬β∨.s↔Kπ`h+π;∨;↔I1ε;⊃1π+;∪↔∩β≠π[␈∪π3*β∂'K∨+7OS∞s∂↔Mbβ↔[↔rβ¬β∂}kC3↔&(4+πw≠←↔Iπ#=βSF)βGW/≠S'?rβ?→β&C∃β'w#↔;ON{9β?2β¬β∨O3↔9βπ∪↔∪'≡S∃9∩)D4(Ph(4(LK→βSF)βOS↔+∂SW⊗)β?→π##∃βn∂#'v)β'Mε[;?←raβ?;*β∂π9ε;'[∃ε9βπ"β#?hQ∃K≠O∪OQ↓ε{K∪↔∩↓βOS↔+∂SW⊗1↓β&+≠';O#'?9+	9↓↓ααS#'~β'Mβ
βCK↔&K∂πS*Z	#~cA%hS←#↔⊗)[Mπ∪↔CK/≠↔;S~β¬βO&S∃β}1βS#*β7π∂FK;∃β∞s⊃[αβK↔C⊗+O↔;'→β¬β≡+;S↔v≠∀4+Nqβ¬β∨+'Sπ⊗c∃β3∞s∨Wπ>)1βπv!Z	G→3A%εKMβSF)βπO≡+KS'}qβS#∂!β←#.qβS#(h+7π≡C';∃εKMβ'rβOSπ&)[Mbβ'Q6∪↔3'/3↔Mβ&C∃βO.sS↔;≡)[Aph)"SF)β∂?w≠'∪↔⊗S'?w→β?→π##'MπβπC↔∩βπK∃εs↔WS⊗1β'rβ∪↔∂N#';≥π;#↔SF+H4+&yβK↔>K⊃β&C∃β?⊗S↔∂Qε{→β.c'↔→εMβ¬π≠↔;S.s∂∃β␈⊃βS=π+O∃β
β7?∪∞`4+?ε+KπS␈⊃β?Iπ#=βπ&k'Q↓+∪CK?ε{O'SN{;M∃
βπMβ∞∪OSK∞≠Qβ?⊗S↔∂S~βS#π h+∂πrβ∃β⊗+3'↔6+⊃9↓¬##∃βεC↔IεKMβ←⊗KSS↔rβπMβ&C?W∨BβO↔;&+;∂↔~βπK∀hSS#∃ε{+↔∨#Mβ?2β↔3N+→1β↔+Qβ'r↓"7∞≡KS#J↓Ee]2Iα%β6[?IπβK?C␈≠'S'}sM%8hR¬β∨.s↔Kπb[≠'↔≠Q[␈∪∪↔Iπ≠SKW∨#WKπbβ∪↔≠Ns'S'}qβ?→ε∪↔3'.1β←?.c⊃β*β∧4+π∪↔∪'≡S∃4⊃"]2jcM3AJβ←#↔⊗)Z]εKMβSF)↓←␈∪3⊃	εK9β←FK∂!β&C∃β7∞≠#';(h(Z5π;#?O*β↔3N+≠Mβ∂∪∃β'rβGW↔∨#'?9εKMβOO#WπS.!84*Jβ∪=βv{QβO.)β#?:βS=β>K[∃β∨+∂!β
β∪↔≠Ns'S'}qβ?→ε∪↔3'.11βπv 4*%π##';Zβ'QβO→β'7ε{OO'⊗c∃84U##↔K.3?K∃π;∃βS/∪9βSzβO↔∂}s⊃β?⊗#↔Iβ&+≠';O#'?;~p4(4PJ¬βO.≠?;⊃ε{K∪↔∩βOSK.≠SWK∞aβ∪↔6K;'SN{9β?2β↔3N+→β'~β¬βO.≠?;⊂hS?K∪/⊃βCK.#'∂π&)X
E9252∩I9↓0→"]2jb	%β∂≠O↔K'→βS#∂!βS#*β≠'K∨!β?K&+H4+π∪↔∪'≡S∃4⊃β'Mε	↓∨}{⊃	βv{S'?rβ?→β⊗+3'↔2β≠?Iπ##∃βn∂#'v)Z5εK84+&C∃β←␈∪3⊃599↓αF+K∃↓⊗;??⊃∩β7↔πw→βS#∂!βS#*β↔3N+≠Mβ&CπQ4⊃βπO∨∪'↔_h+S=↓25βπ?∪↔∃β>KS!β␈+Iβ'&+πMβ}1β←#∂!β↔fK↔≠M↓25β←␈+3⊃βF[∃1εs?Qβ&CπP4W##∃β⊗+3'↔7→βS#.kO↔36+Mβπ⊗)βSK.)84*&C∃βπFK?7π&KkπSN{;Mβ}1β↔fK↔→βNqβS#*β3'S/∪πSW⊗)βπK*βCπK&Kπ04W≠↔∂?v!β?K&+Iβ∪.3';'&K?;Mph(4(LK9β∨.s↔Kπba↓∃O
βO↔∂}s⊃β?⊗#↔Iβ&+≠';O#'?9ε;'[↔~β∂K'&+K'¬ε3?H4V≠K'SN≠'k'v9βπ9εO∂KOβS'?rβ?→β
βGWπfKSeβ&yβ¬β∨KOS↔jq∃D4U;∃βO.;∨↔O"βS#π"β?SBβ?WIε≠?77}qβO↔w≠∃βπv!βO∂N+;S'6K
βW≡∨∃β}04+;␈!7∪'⊗+∂S3Jk?O/∪[πf)βGW∞c'S'/→β∂?↔∪↔OC}s∪Mβn{K∃βf{O↔3JβS=β≡+∂?; h+?K&+IβO'∪W∂S/∪π1β&+≠';O#'?9π##π9π#=βπwIβ/'v!β?→ε∪↔#π6K?Kπbβ∪↔≠Ns'S'}q84*v{S∃β&CπQ↓ε	↓βO.≠?;⊃ε{K∪↔∩↓β∪↔6K;'SN{84+≡;;?"↓β∨W∂∪π;S.)↓βSFQ↓β&C↔K∃ε+c'O"↓βCK.#'∂π&+M↓↓+∩	∃Eαβ7↔↔&K;≥β&C∀4+∨∪'S↔⊗K?9~β?Iβ&CπQβ∨+∂!β
↓∃J	+	β'Mπ+;'G.)84*≡{7∃β∂+π3'&K↔Mβ∂∪∃β/≠Qβ∪.3';↔"β+?'w#3eβ>KS!β⊗+3πS. 4+G.3'SN+M1β*s≥9β⊗+3'↔7→βπ;"β∨?πg→β7πJβK↔G.KK∃βV{';Qπ#K↔π&k↔;Qph(4(M≠↔∂?v!↓↓β␈∪∪↔Iα↓β∪↔6K;'SN{;Mβ∨∪'S'ze   whole   belief
structures rather than  individual beliefs.  We can  treat individual
beliefs  by  saying  that a  system  believes  %2p%1  in state  %2s%1
provided all "reasonably good"  %2B%1's satisfy %2B(s,p)%1.  Thus  we
ar↓JAI%giS]≥kSgQ%]NAi!J@ES9iKeg∃GiS←8DA←L↓iQJAIKCg←9CEYr↓O←←HJe∧JDOf\~(~∀∩Q¬\AC]¬Y←Or↓oSiP↓GesaQ←OeCAQrA[¬rAEJ↓QKYa→kX\@~∃/J↓g←Ym∀ABAGIsai←≥eCZA	rA[C-S]NA!sa←i!KgKf↓CE←kPAiQJ↓giek
ikeJ↓←LAi!J~∃G%aQKd↓C]HA¬E←kh↓iQJAQeC]g1CiS←8A←LAACeif↓←LAi!JAGSAQKdAQKqh\A∨kd4∃g←YUiS←\↓SfAG=[aYKQJAoQ∃\AoJ↓QCmJEOkKMgKHD↓BAGSAQKdAMsgiK4AiQCPAae←⊃kGKf4∃iQJ↓GesaQ←OeC4AMe←4ABAa1CkgS	YJAa1CS]i∃qhA[∃ggCO∀\@A)!←kOP↓oJA]∃mKd~)ae←m∀AiQCPA←kd↓g←YkQS←\A%fAk]%ckJX↓io↑A⊃SMMKIK]hAM←Yki%←]fA¬eJ~∃¬Y[←gPA]Km∃dAM←U]HAKaGKah↓M←dAYKerAMQ←eh↓GesaQ←OeC5f\@A%\AiQ∀AC]C1←OrX4∃iQJ↓gKG←9HA←e⊃KdAI∃MS]SQS←\@AG←eIKga←9IfAi<AiQJ↓OK]KICXAS⊃KBA←_~∃K]
SaQKI[K]h0AC]H-∧ASLAiQJ↓aCei%GkYCHAgsgQKZAkMKH\@↓/QSY∀AoJA]SYX~)eCeK1rAEJ↓CEYJ↓i↑AaI←mJAU]Sck∃]Kgf0AoJA⊃←\Oh↓KqaK
hAi↑↓MS]H↓io↑@∀e∧JcL~∃E←QPAgCQSgMs%]N@ε$\~∀~(∪∪h@↓gKK[L@Ai↑A[J@↓iQChAiQKIJAgQ=kYH@↓EJ@A∧@A[KQCiQK=eKZ@↓←L~∃5CiQK5CiSG¬X@AY=OSFA¬ggKeQS]N@↓iQCh↓]←hA¬YX@AMKG←]⊂A←eI∃dAIK→S]Si%←]f~)GC\@↓EJAe∃IkGK⊂@Ai↑AMSeMh@A←IIKdA⊃KMS]%iS←]L@AC]⊂@AMkIiQKdAiQK=eK[f4∃GQCICGiKISuS]≤@AiQ=gJ@AMKG←]⊂@@A←IIKd@↓IKMS9SiS←9f@Ai!Ch@A¬I[ShAgkG ~∃eK⊃kGiS=]f\@↓'kGPAiKG!]SGC0AeKgUYifX↓SL@AQQKrA
C\AE∀@AM←U]HXA5Cr@A	J~∃Q∃YaMk0AS\@↓aQSY=g←aQdAC]H↓S\@AQQJAG=]gieUGiS←8A←L@↓M←e[¬XAgG%K]iS→SF~∃QQK←e%Kf\@↓∩Ao←UYHAG=]UKGQkeJAQQChA5C]rA=LAiQ∀AS]M=e[CX↓aQSY=g←aQ%GCX~)CeOk5K]if↓iQCh↓GKei¬S\A[∃]iCXAG←]
Kaif↓GC]]=hAEJ↓eKIk
KHAi<@AaQegSGf4∃oSY0Aike8A←kh↓i↑AE∀AgWKQGQKf↓←LACIOk[K9ifAi!ChAi!KgJA
←]GKAifAe∃ckSe∀~∃gK
←]H@!←dAQ%OQKd$A←eI∃dAIK→S]Si%←]f\4∀~∀∪!KeJ@↓SfAC8ACaaI←qS[¬iJAg∃G←]H↓←eIKHAIKM%]SiS=\A←L4∃EKY%KL\@↓
←dA∃CGPAMiCiJJefJDA←LAQQJ@A5CGQS9JAC]⊂~∃KC
P@Ag∃]iK]
J@-`↓S\@A∧AgkSQCEYJAYC]≥kCOJ-_X~)oJACMgSO\↓ieki Ai↑@∀e∧Qf1`RJb↓SLAC9HA←]1rASL4∃iQJ↓[CGQ%]JASLAG←]MSIKe∃HAi↑↓EKYS∃mJ@JI`Jb@↓oQK\4∃Sh@↓Sf@A%\Agi¬iJ@@∀efJb8@@@AQQJ@A1C]Ok¬OJ@@∀e_Jb↓Sf@A
Q←gK8@AM←H@A←kH~∃G←9mK]S∃]GJX↓C]H@↓iQKe∀ASfA9↑ACgMk[ai%←\@AQQChAQQJA[¬GQS]∀AKqa1SGSi1r~∃e∃aeKg∃]ifAMK]iK9GKfA=L@Je0JbAS8AC]r↓oCr\A)QkLAoJA
C\Ai¬YVAC	←khAQQJ~∃	KYSK→fA←LAπQS9KgJXAI←OLXAG←Ia←eCQS←]f0@AiQ∃e[←gQCifX↓C]H@↓G←[aUiKd~)←aKe¬iS]NAgsgQK[f@↓oSiQ=khACMgk[S9N@Ai!Ch@AQQKrAUgJ@A∃]OYSMP@A←HA←kd4∃MCm=eSiJ↓MSegPA←eI∃dAYC9OkCO∀\@@-0A[Cr↓←dA[¬rA]←PAEJAQQJ@A1C]Ok¬OJAE∀~∃iQ∀AYC]≥kCOJ↓oJACIJAkg%]NAM=dA[C-S]NA=iQKd↓CggKIiS←]LXAJ]≤\AoJ↓G←kY⊂X~∃oISiS]≤AS\A∃]OYSMPXAgegiK[¬iSGC1YrAkMJA
e∃]GPAMK]iK9GKfA¬fA←E)KGif↓←L~∃	KYSK_\@A⊃=oKmKHX@Ai!JAEKMhAGQ=SGJ@↓M←dA¬eiSM%GSCX↓S]iK1YSOK9GJ@A]←eV~)[CrA	JAi↑A[CW∀@-_A∧AgkEMKhA←_A←kd@E←kQKdDA1C]Ok¬OJAe∃gieS
iKHAM↑ACf4∃i↑A¬m←SH↓iQJAACeCI=qSGC0AgKY_[eKM∃eK]G∃fA←LQ≠←]QCOkJbrlf$\~∀~(∪/JA9←nAgUEUKGP@-∧QLY`RAQ↑AGKIiCS\↓GeSi∃eSBv↓R]J\ε,Q∧1.R@~)SfAG=]gSI∃eKHAQekJAAe←mS⊃KHAi!JAM←1Y←oS9NAG←9ISiS=]fACIJAgCQSgMS∃Ht~∀4∀∩L\↓)QJAMKh@JI¬KXQLRJbA=LAEK1SKMf0AR]J8AiQJ↓gKhA=L@Je@JbOf↓M←d~)oQSG @@JeλQfY`$JbASL@ACgMSO]K⊂Aiek∀@AoQ∃\@-~↓SfAS8AgiCQJ@-f~∃G←9iCS]L@Agk→MSGS∃]iYrE←Em%←kfD4∃G←]MKckK9GKfA=LAg←5JA←L↓SifA5K[EKIf\~∀4∀∩L\@@Je	KXQf$JbAG!C]OKLAS\A∧@AeK¬g←]C	YJ@A]CrAo!K\Ai!J@AgQCiJ~)GQC]≥KfAS8AiS[∀\@@@↓/J@@↓YSWJA]KnAEKY%KMf@↓i↑@A	J@AY=OSGC0@A←d@EaY¬kgSE1JD~∃
←]gKEkK]G∃fA←L↓←YHA=]KfA=dAi↑↓G←[J↓S\ACL@JeG=[[k]%GCiS=]fJbAS\AM←[J~)YC]OUCOJA=\AiQ∀AS]aUhAYS9KfA←H@Ai↑↓EJ@JI←EgKImCiS=]fJb0AR]J8@AEK1SKMf4∃CE←UhAiQ∀AK]m%e←][∃]hAi!JAS]→←e[CQS←\A→←dAo!SGPA
←[Kf↓S\A←8AiQJ↓S]akP~∃YS9Kf\@↓)QJAMKhA←_AEKY%KMfAMQ←kY⊂A]←h↓GQC]≥JAi←<AeCa%IYrA¬fAiQ∀AgiCQJ~∃G!C]OKLAoSi AiS[∀\~∀~(∩L\@A/JAAeKMKH@AiQ∀AgKhA←LA	KYSK→f@Ai<AEJ@↓CfAG=]gSgQK]h@↓Cf~∃A←ggS	YJ\@QβI[%iiKI1rXAG=]gSgQK]GrASfA9←hAB↓ckC]QSiCi%mJAG=]GKaPAS\~)[CiQ∃[CiS
CXAY=OSF@4@ABAMsgiK4ASf@↓KSiQ∃dAG←9gSgi∃]hA←H@A]←PXAEkP@ASh4∃o←k1HAgK∃ZAiQ¬hAoJAoSY0Ag←[∃iS[KLAQCm∀Ai↑A¬gGeS	J@AS9G←]g%giK]PAgKiL~∃←L↓EKYS∃MfAi<A[CG!S]Kf↓C]HAAK←aY∀\@A∨UdAS]QkSiS=\AgCefAiQ¬hAoJ↓gQ←k1H~∃E∀ACEY∀Ai↑A5CS]i¬S\ACIKCfA=L@AG=]gSgQK]Gr↓S\A←UdAEK1SKMf↓C]HAQQCh@↓Sh~∃5CrAE∀@AKgAKGSC1YrAS5a←ei¬]h@AQ↑ACm=SH@A%]G←]MSgiK9GSKf↓S\@AQQJA[¬GQS]∀Of~∃AkeKYdAC]C1siSF↓EKYS∃MfR\4∀~∀∩_\@A∨UdAGe%iKeS∧AM←d↓EKYS∃LAgsMiK[f↓GC\A	JAgiIK]Oi!K]KH↓SLAo∀~∃SI∃]iSMdAg←[∀A←LAQQJA[¬GQS]∀OfAE∃YSKMLACfA∃qaeKMgS]N↓O←CYLXAR]∀\ASL↓oJ~∃!CmJA	KYSK→fA←L↓iQJA→←eZ@	∪hAo=kYHA	JAO←=HASL\\\D8@A)Q∃\AoJ↓GC\A¬gVAi!Ch~∃QQJA[¬GQS]∀OfAE∃QCmS=dAEJ↓g←[K]QCh@∀eeCi%←]CX∀bXAR9J\@@∀eShA⊃←KfA]QChA%h~∃E∃YSKm∃fAoS1XACG!SKmJ↓SifA≥←CYf∀b\A)!JA[←IJA←L↓SifA	KQCm%←dAo∀AGC\↓CGG←U]h~∃→←dAS8AiQSLAoCr0AiQJ↓EKii∃dAoJ↓oSYX↓YSWJ↓iQJA→k]Gi%←\@JI∧QfY@RJb\A/JA¬Yg↑~)o←kY⊂AYSW∀Ai↑AIKOCe⊂AS]i∃e]CX↓giCi∀AGQC9OKfA¬fAGQ¬]OKf↓S\AE∃YSKL↓S\~∃M↑AMCHACfAQQSfA%fAeK¬g←]C	YJ\~(~∀∩L8@A∪L↓iQJA5CGQS9JAG←5[k]S
CiKf0AR]J8AK[SQfAgK9iK]G∃fAS\↓g←[J4∃YC]≥kCOJAiQCP@AGC8@AEJAS]i∃eaeKQKH@A¬f@ACMgKei%←]fXAckKMiS←]L@AC]⊂~∃G←5[C]ILXAoJ↓oSYX↓oC]h↓iQJA¬ggKeQS←]f↓i↑AE∀AC[←9N@ASQfAEK1SKMf↓k]YKMf~∃o∀ACeJACgGISES]≤Ai↑A%h@AB↓O←CX↓←d@AMkEO←¬XAiQ¬hAS]Y←YmKL@AYs%]N\~)/JAo%YXAE∀A[←gPAgCi%gMSK⊂AoSi A←kd↓EKYS∃LACg
eSai%←\XA%LAoJ↓GC\A¬GG←k9h~∃M=dASiLAG←[5k]SG¬iS←]LACfA→keiQ∃eS]N↓iQJA≥←CYf↓oJACIJACg
eSES9N\~∀4∀∩L\A'←[∃iS[KLAoJAMQCYX↓oC]h↓i↑ACMGeSE∀AS]iI←gaK
iSmJ↓EKYS∃MfX~)J]N\↓BAEK1SKLAQQChA%hAI←∃fA]←PAW]←\AQ←n↓i↑AM1rAi↑↓¬←gi=\A←d↓KmK\↓iQCh4∃ShA⊃←Kg\≥hAW]=nAoQ¬hASh↓oC]iLAS\A∧AGKeQCS\AMSikCQS←\\4∀~∀∩_\A
S9CYYr0AoJA]SYXAAeKMKHABA[=eJAK
←]←[%GCXA¬gGeSAiS←\Je∧JD~∃i↑↓BAYKMfAKG=]←[S
CXA←9J\@AQQJ@A→KoKd↓EKYS∃MfAo∀ACgGISEJA¬]HAi!JAYKMf~∃i!KrAG!C]OJ↓oSiP↓giCi∀AG←]MSgiK9hAoSQPACG
←k]i%]NAM=dAiQ∀~∃EK!CmS←HAC]H↓iQJA%]iKe9CXAgQCiJA
QC]O∃fXAi!JAEKQiKdA]JAoS1XAYS-JASh8~∀~∀%)QJA¬E←mJAGeSQKeSB↓QCmJAEKK8@AM←I[kYCQKHAg=[KoQ¬h@Am¬OkKYd\~∃)!Sf@A]←kYH↓EJAE¬HASL↓iQKe∀@AoKIJAoS⊃KYrA⊃SMMKIK]hA¬gGeSAiS←]LA←L@↓EKYS∃Mf~∃Q↑ABAACeiS
kYCd↓[CGQ%]JAi!ChAC1XA[KPA←kdAGeSQKeSB↓←dAS_AiQJ↓GeSi∃eSB~)CYY←]KH@A¬gGeSAiS←]LAiQCP@AIS→MKeK⊂AoSI∃YrAMI←Z@A=kdAS9ikSi%←]f\@A≠r4∃aeKMK]hA=aS]S=\ASf↓iQCh↓[←eJ↓iQ←k≥QhAo%YXA[¬WJAi!J@AGISiKe%BAg←5KoQCP~∃[←IJAae∃GSgJAChA9↑AG←Mh@AS8ACaa1SGCE%YSir0AEkh↓iQChAiQKd@Jeg!←kYH∀b~∃gQSYXAIK[CS8@AeCQQKdAYCOkJ0AR]J8AoJAMQCYX↓oC]h↓i↑ACMGeSE∀AEKY%KLAS8AB~∀∀eMC[%YrJb↓←LAG¬gKf\4∃⊃←o∃mKdX↓KmK\↓ChAi!JAae∃gK]h↓YKmK0A←LAYCOkK9KgfX↓iQKe∀Aae←	CEYr4∃o←\≥hAEJ↓eCIS
CYYr↓ISMM∃eK]h↓KckC1Yr@E≥←←HD↓CgGe%aiS←9fA←L↓EKYS∃LAM←H~∃gsMiK[f↓←LAaICGiS
CXAS9iKeKMh\@A%LAiQ∃eJAo∃eJXA]JAo←UYHA]=iSGJ4∃k]e∃g←Ym¬EYJA¬[ESOUSiSKLAS\A=kdACMGeSaQS←]f↓←LAE∃YSKL↓i↑A←UdACGEkCS]QC]GKL\~∀~(∪/QS1JAoJ↓[CrA9←hAo¬]hAi<AaS\↓I←o\↓←kdA≥K]Ke¬XASI∃BA←L↓EKYS∃L~∃i<ABAg%]OYJ↓CqS←5CiSu¬iS←\0AoJA]SYXA9KKHAQ↑AEk%YHAaIKGSg∀ACqS=[CiSiCiS←9fA←L4∃EKY%KLAC9HA←i!KdA[∃]iCX↓ckCY%iSKf↓S]i↑particular intelligent computer
programs.


#. %3Definitions relative to an approximate theory%1.


	Certain  concepts,  e.g. %2X can do Y%1, are meaningful as
statements in rather complex theories.   For  example,  suppose  we
denote the state of the world by %2s%1, and suppose we have functions
%2f%41%2(s)%1,...,%2f%4n%2(s)%1  that  are  directly  or   indirectly
observable.  Suppose further that %2F(s)%1 is another function of the
world-state  but  that  we   can   approximate   it by

	%2F"(s) = F'(f%41%2(s),...,f%4n%2(s))%1.

	Now  consider  the counterfactual
conditional sentence, "If %2f%42%2(s)%1 were 4, then %2F(s)%1 would be
3 - calling the present state of the world %2s%40%1." By itself, this
sentence has no meaning, because no definite state %2s%1 of the world
is  specified  by  the  condition.   However, in the framework of the
functions %2f%41%2(s),...,f%4n%2(s)%1 and the given approximation  to
%2F(s)%1,  the  assertion  can be verified by computing ⊗F' with all
arguments except the second having the  values  associated  with  the
state %2s%40%1 of the world.

	This gives rise to some remarks:

&. The most straightforward case of counterfactuals arises when
the state of a phenomenon has a distinguished Cartesian product
structure.  Then the meaning of a change of one component without
changing the others is quite clear.  Changes of more than one
component also have definite meanings.  This is a stronger
structure than the %2possible worlds%1 structure discussed in
(Lewis 1973).

&. The usual case is one in which the state %2s%1 is a substantially
unknown entity and the form of the function %2F%1 is also
unknown, but the values of %2f%41%2(s),...,f%4n%2(s)%1 and
the function %2F'%1 are much better known.
Suppose further that %2F"(s)%1 is known to be only a fair approximation
to %2F(s)%1.  We now have a situation in which the counterfactual
conditional statement is meaningful as long as it is not examined too
closely, i.e. as long as we are thinking of the world in terms of
the values of %2f%41%2,...,f%4n%1, but when we go beyond the
approximate theory, the whole meaning of the sentence seems to
disintegrate.

	Our idea is that this is a very common phenomenon. In
particular it applies to statements of the form %2"X can do Y"%1.
Such statements can be given a precise meaning in terms
of a system of interacting automata as is discussed in detail in
(McCarthy and Hayes 1970).  We determine whether Automaton 1 can put Automaton
3 in state 5 at time 10 by answering a question about an automaton system
in which the outputs from Automaton 1 are replaced by inputs from
outside the system.  Namely, we ask whether there is a sequence of
inputs to the new system that ⊗would put Automaton 3 in state 5 at time 10;
if yes, we say that Automaton 1 ⊗could do it in the original system
even though we may be able to show that it won't emit the necessary
outputs.  In that paper, we argue that this definition corresponds
to the intuitive notion of %2X can do Y.%1.

	What was not noted in that paper is that modelling the
situation by the particular system of interacting automata is
an approximation, and the sentences involving
⊗can derived from the approximation
cannot necessarily be translated into single assertions about the
real world. 

	I contend  that the statement,  %2"I can go  skiing tomorrow,
but I don't intend to, because I want to finish this paper"%1 has the
following properties:

	1. It has a precise meaning in a certain approximate theory of
the world
in which I and my environment are considered as collections of interacting
automata.

	2. It cannot be directly interpreted as a statement about the world
itself, because it can't be stated in what total configurations of the
world the success of my attempt to go skiing is to be validated.

	3. The approximate theory within which the statement is meaningful
may have an objectively preferred status in that it may be the only theory
not enormously more complex that enables my actions and mental states to
be predicted.

	4. The statement may convey useful information.

Our conclusion is that the statement is %3true%1, but in a sense that
depends essentially on the approximate theory, and that this intellectual
situation is normal and should be accepted.  We further conjecture that
the old-fashioned common-sense analysis of a personality into %2will%1
and %2intellect%1 and other components may be valid and might be put
on a precise scientific footing using %2definitions relative to
approximate theories%1.

	If, as we conjecture, most common sense and even scientific
terms are meaningful only in approximate theories, then a philosophical
method as old as Socrates needs to be re-examined.  This method
involves attacking a common sense notion by introducing examples that
have not been encountered in the previous usage of the notion and
showing that in these cases the notion gives unacceptable results.
But it may be that any theory of ⊗just or ⊗unjust actions must
be based on an approximate model of the world, and anomalies can
always be found.  The philosophical analysis cannot invalidate a
notion by finding limits on its applicability; that requires find
a better and more general notion.  This may not always be possible,
and even when found, the new notion will still be limited.  For
example, there may never be a notion of just actions securely
founded on quantum physics and chemistry.  Moreover, this conclusion
does not depend on any considerations of ⊗emergent ⊗phenomena. 


.SKIP TO COLUMN 1
.ONCE CENTER
.bb EXAMPLES OF SYSTEMS WITH MENTAL QUALITIES

	Let us  consider some  examples of machines  and programs  to
which we may ascribe belief and goal structures.

.ITEM←0;
	#.  %3Thermostats.%1   Ascribing beliefs to simple thermostats is
unnecessary for the study of thermostats,
because their operation can be well  understood
without it.  However, their very simplicity  makes it clearer what is
involved  in the  ascription, and  we maintain (partly  as a
provocation to  those who  regard  attribution of  beliefs  to
machines  as mere  intellectual  sloppiness) that  the ascription  is
legitimate.$

~Whether a system has beliefs and other mental qualities is not
primarily a matter of complexity of the system.  Alt↓Q←UOPAG¬efACIJ~∃[=eJAG=[aYK`AiQC8AiQKI[←gi¬ifXA%hASf↓QCeH↓i↑ACMGeSE∀AEKY%KMfA=d~∃O=CYfAQ↑AiQ∃ZXAC9HAiQ∀AgC[∀ASfAAKeQCAfAieUJA←L↓iQJA	CgSF↓QCeI]CeJA=LAB~)G←[aUiKdX↓R]J\↓iQJAACehA=LAiQ∀AG←[AkiKd↓iQCh↓KqKGUiKfAQQJAaI←OeC4~∃oSQQ←kh↓iQJAAe←Oe¬ZASiMKYL\4∀4~∀%
Se`∨!↓β∂}sO'∪/⊃β¬β≡K7C3*↓βS#/∪7?O&QβSFQβS↑&w~
|f2πMPhV\↔"π⎇VrπMRαπL]Wε/,≡G/⊗T
↔~ε∀F.?,\Rαε≤-w6*∞Mε*πL]Wε/,≡G/⊗T∞6/"∧
vph.Mε*πMW⊗n}>F∂"D∧π'/-n2ε}d∞FF*
V∂"∧∞vF.d∞FF*∞LVoε↑,↔'/,TαεO4∩ε&\}&.(Q,&.f}tπ&FTF/≡≡Y,Dλ≥→-↑→<X.N<Y+∧;Yλ
L8=Y.4λ≥~T~→8.D_<↓Pλ4yP;Z2w⊂⊂≥42FE≥2vx2\0z:y→P⊂4yH⊂⊂4wλ⊂:42H⊂:8πo   degree  range  around   the  desired
temperap	UeJ\AQQJAg%[aYKMhAEK1SKLAAdβ↔∪N≠πS∃α)J	#~cA%∃
βπO∂⊗K↔Mε∪↔3'.04+Szβ?;3↓∀π&G,\Rπ≡\β]]Xy<g$λλU
(≤[m⎇(~<d∞7wP_wv2⊃⊂⊃*4→P97w[P4yPλ:oo
hot", and "The room is OK" - the beliefs being assigned to rβiC`&+Hλh-|bαπMRπ ~→<M]|⎇_.D~;H∞M→(≠l.Z;⎇.4≥x>%a"Uz]H≥~Q"\α4→y6wy]0z⊂1→v4r{→yP⊂:~2P⊂9≠wvP4\P⊂:7[P1wv→⊂⊂7yλ:oo  hot, it sends  a
message saying so to the furnace. A slightly more complex belief
predicate could also  be used in  which the thermostat  hapε↓B@AE∃YSKL4∃CE←UhAoQ¬hAiQ∀AiK[AKeCiUdβ∃β≡C?W3"β∃β∞s⊃βπv{S#↔∩β↔3N+→βπ⊗{WQβ>CπQβO 4+'~q↓↓αO!β'Mεs?Qβ≡c↔πIαβ←#'≡Aβ'Mε∪↔SS/⊃1β↑BεNd∧π>*∞⎇↔≡F\Dπ&z=vw≡≤LW⊂h.
w∨≡≤-F*ε↑.&␈↔4
⊗rπMRπ&↑&n}\↑F/∩D∞FF.d∞v*π⎇}Vf"≡6∨⊗≤,PhV,]FN.n4αε∞-}W"π⎇↔"πMRπ&]↑ε/⊗≡NW⊗*
≡2r¬|Tαε&t
f␈"≡6∨⊗≤,PhWMtαεOD⊗wJ
}FF/$∧ε⊗.M≤V7≠4
↔"ε≡2αεmtε␈ε≥m⊗}r↑f.r∧⊗⊗␈↑Dπ>F↑Mε/∩∞Mε(h-V∂"
≡2ε}d
w∩ε|lbε␈$⊗⊗␈↑Dπ&FT∞v.∂MW∩ε}$ε∞⊗}↑Bπ>
tπ>}d∞FF*,↔' ≠→(
|C"Ul≡→<[
⎇kHλ∧∧∪;|L]⎇Y<ED~=λ∧
_<h∧
[h~-n≤[|n8⎇~.l(λ_L]~99N5λλ~%l+Hλ∧
=β"LMy<{D}λ_Y-M9=Y$∞~_=∧
=λ_L]~9=L↑h≥~T≤[{mT~<h∞M{h~
}C"AQB5~T≥→;.<X=∞↑Y(_m⎇]≤[mD≤}<nL;(~-dλ≠>$
≠⎇<lT≠8>$Y(λL<x|M≤Y9β!,<hλm{≠≠n}nH∃
<[;n>_=≤d∧≥<≤nL:<\d;Yλ∧≠⎇{N>_:<N4λ≥→-Mλ≥~Tλ_y-n≤X;↓Q\}<nL;(λ∞Mh≥≥.-Hλ≠md≠|H∞=≥=λ∧
yYH

⎇λλ∞|=→<D[≠⎇d∧≥≠h∞M→<y$<Y8.5Hλλλ⊃"Xy-n≤X;∧∞x=→.%=→;.<X=∞↑Y(≥
<[;n>_=λ∞L;≠≤d∞~→(n<[X,<(≥≠d∞≥<[D∧≠{H
}C"[llHλ≥
∞<hλ
<9<~-lh≥~Tλ_y-n≤X;∧∧~≠⎇∧∧≥x=↑Hλ≤L↑y<]M⎇<H_.Dλ≥~Tλ≤Z,⎇≥β"NL;<→..≥<Y%dλ∀Y,<;]≠∂∀~=λ∞|<h≥
⎇h~≠nD≥<≤nL:<\eD_;Y∧∞~→(∞≡9<⎇
≥{H_.-|y#!,<h≥
t≥z→.M→<H∞M→(≥.∞⎇_:..h≥~↑[;|nL=λ≠-≡⎇_:l][≡(∧∧,XY-M9=Y,D,(~.D≥x<aQ]≠{d∧_{{Dλ≥<∞>_:<N4λ≠|D∧≥z→.M→<H∧∞~→(n<[X,<(λ≥
<[;n>_=λ∧
:<⎇≥y;[∂⊃"I,L,;~9.l9λ	&∀≥~→$∧≥x=↑H≥x.4≥≠{d∧_{{EHλ∩.Dλ≥≥.-Y9λ
}=λ≥
=λλ
l:=~↑C"[-≡⎇_:lT≥x<d
89→'4≥~→$≠⎇{N>_:<N4_{{NN[{≠↑H	,NNZ99∧V(≥≠d∞≥<[D
yYH∞M→#"Lm≠⎇h∧
yH≥l≡→<H.=λλ∧VX{⎇-L≠I⎇∧V+λ_L\x=<lT≥~→$∧≥X;∞l(≥x.4λ≤⎇∞\zkH∧
~→#!.≠≥;,,<H_l≥9(λ
⎇Xy(∧;Yλm⎇;Y∧∧≥~→$∞≤[⎇,-→+λ∧;Yλ∧x;9$9x:-dλ≥z]Hλ_!Q\Y<
L8y;,]]λ≥L≥≥Y(∧∞x<h
}Y→<L\Hλ∧
z;XlT≥~→$∞y<]M≤y<h
|Hλ≤
N;8Y..h_<LQ"Z;L>Y8<m≥Y{≡$∧→>≤]\z=LUλ_;LDλ≠:,>[x{m↑≥=→..h_<LTλ~;L>Y8<m≥Y{≡$z→8.¬β"[ml(~<d
→9λ∞Mh→→.=9{H∀≥→;.<X=∞↑Y(_m⎇]≤[mD≤}<nL;(≥
=λ≥m};→λ∧VZ{[nt,#"L∀≠≠⎇∧
;|Y$8[⎇.D≥~→$∞~→<M\;λλ∞>_=→$
yH≥
(~≠n↑y(_-lλ~=∞4≠⎇{D∞⎇_=T≠yC!-→8;∞MC"AQB2;D∞~→(m<\⎇∧∧≤≠_,<+λ≥m
;→(∞M→(λ∞?<⎇→-T	,Xm};→≠D}	,(∞N<[H∧
yYC!.~→(m≠⎇h
|H~≠nD≥x=↑Hλ≥.∞⎇_:..kλ≥
<Y(
≡h≠[d∞Y8<m⎇Hλ≥
t_<xn-8Y(∞Mc"Z.D≥~→$∧	,Zmm⎇{→,Ly),$∞~_=∧
=λ_m};→≠D}λ_-lλλ	&,(→[n.~;|M∀,(~.D~_9∧
[c"L≤Z;~.O(≥≠d∧,X{m]=;Z,<=→)&∀≥~~.4	,YL≤⎇	,$∧≠|H∞Mh≥_-<(~=∧
;]≠d∧_8xm};]β!-;H_m⎇]≤[mM~;Yd∞~→(∞?<⎇→-UHλ⊂$
;|Y$9≥X-ly9λ∞?<⎇→-T≥{⎇-Lλ~{M}h≥z↑~→<AQ]~→$∧	,X,>~;{N4,(~.D	,X.N→;<∞L9	,$∞⎇8xl\9→9¬D_;Y∧
=λ≥m};→λ={;=-m8x=Q"YX-≥≥<Y.4_;Y∧9_<∞Dλ≥≠d∞~→;%dλ
∃lTλ_9≡≥→9∧∞≠h≥
(→X-≥≥<Y$∧_↑(∞N<[Z-lc"[llHλ≥
(≥z
⎇→(λ∞?<⎇→-Tλ≥;NM;λ≥
(λ≥m
{→(∧
≠⎇<lT_{{mL9λλ
|YH_-lλλ≥
;C"ML=≥~-lh≥~T≥≥{d∞_<]∞4≥x<MT≥<λ∧∞≠yy.M→<KD∧∃~→$∞≤Y<l]]λ≤o≡⎇→;$
_<h∧∞~→#!$,\~∂≡z8x-Dλ_x.8Z;
≡≡),$∧≠yH∧≠z;Ltλ≥~
≡hλ→.l;Hλ∧
9Hλ
≡λλ~≡{I⎇∧∧≥~→!QI,Zmm⎇{→,Ly),$
|H≥
(	,N⎇;≠	&∃C"KN=z<λε!"C"A∀kH	&:y;→E↑Y<≤M|≥8z-lhλ~-n→;≠
≤y;]∧∧_{{Lm9⎇<L≡~;{N4λ~;D∧_(λ<;≠≥-L<C"L≡=≠{,≡≠{H∞⎇|[→¬d,(λλ∀¬Xy-M≥;_.$¬X=.M{8=
⎇H≤}.>→;(≡|z9mnh≥≠d88z∧∞Y<]←β"Z-dλ_(∧y<]≥;H→n,<~λ∀→Z;M≡→(_.↑≠{8.M{KH∧
~→(∞>_=→$
yH→,≤zλ_.↑≠{8.M{C"L≡λ≥~-\(¬]¬6(→→.;Y≤d
{H~.Nh≤⎇≡→(_.D≥~;,T	,]∧V(_;LD≥~→$∞⎇_=↑hλ≠la"Z=∞4λ≠Y-≤z_[n.hλ_.Dλ≥~-\(	,ND,+H
M→(≠-}⎇λ_m⎇;;{D|X<
∧~<h∞M→(_..X>(
|C"\
⎇;]≤dαJ≡∂∃(~;D∞~→(∞
_;Y$∞z=~∧
;]→,|<H_mu;|Y
≥X=→.4¬↑λ≥Yλ¬O∃Hλλ
M→#"Lm<\⎇∧∧≥<y$
yH_l]≠≥;≡H_=.M{8=∀≥x<d↑(≥M⎇H∪Y.]8;[D¬.-Gu(≥z
t→[⎇-lλ_#!&Mh≤nL=→(≡=≠{,≡≠{H∞M_=λ=⎇;→∧Y(≥.<9λ≥
tλ_{mn⎇≤],>λλ≤l]→K<L↑≤[y∞\z;YaQX{{Lm9⎇<L≡~;{D∧λ≥~≡λλ≥l↑Y(λ≥≤{h∧∞;Z=L↑\x;∧∧_{{.∞=→<N5Hλλ∧
~→(∧X<z,1"X=.M{8=
⎇H~;D∞[{H	l=;8-mI|h∞?<⎇→-T~_9∧(→~.>~;Yn]<z→,D≤⎇_.L(_x-M→9λ∧εβ"X-lλλ_$∧≤≠z-nλ~;D∞⎇_=Tλ≥m
|y(m⎇<H
l:9z-|\h∞|<Y(≥≤{h
≥H≥~≡λ≤⎇≡→#"N⎇⎇;→∧∞Y;8-≥H~;D∞⎇_=TH∧
~→(
≥Z=~,≥λ_{mlZ9⎇.,=~;mnhλ_m⎇\z9↑Y9λ∧
_9β!,;≠λ∧]=λ∧(λ→M≥Z=→$∧≠];,,<H≠ld_y;
Nh~;D∞⎇_=Tλ≥Yλ
|H_{n↑\y+∧∞~~<aQ\≤[n<]≡$∞{⎇;D≤→<N=<⎇λ≥≥~≠n\zλ≥
(≠]-\Y<H
|Hλ≠M⎇K>Y.-hλ_l]≠≤h∧
:9z∞A"Y|M}h~;LL9Z;M≡→;≡$∞z=~∧∞~;9%a"C"A~~→(∧∞y;→E↑Y<≤M|≥8z-lhλ≤o≡⎇→;$∞<y9∧∞~→(∞>_=→.4≠yH∀≠≠{Lt≤⎇≤M≡β"[ld≠[{E←Y<[dy;≠∞4_<h∀λ]_.(H_m⎇]_:-m;Yh
≥\⎇≤N\⎇~;mnh≥≠d(λ]-m=Y<N<;β"L={\⎇∞.8⎇≠n$Hλ_m⎇YZ9n↑X=~-⎇Hλ≥
=λλ∞⎇⎇;→∧∧_{{N>≤]8nDλ_(∧{|≡$∧≠yH∧∞~→#!,{{YM≤⎇<X.M;{H∞Mh_Y$∞Y<≤M|≥8y,D_]=∧∞z=~∧88z∧y;≠∧
;H_$∞_<|m≡Y(λ∞>_=→!Q]~_.Dλ≥{n]→λλ∞<\z.>λλ_.4λ≠≠mlhλ_.4~=≤d
Y:9m[|\d∞y<Y$;≤{d
;H≤≡|z=LQ"\⎇≡→<kD∧⊂9]↑H≥~T_{{N>≤]8nM;{H∞
_<y%D≥~→$∞_<→$∞{⎇;Dλ_Y$∧_{|
≤9λλ∞Mc"[,≥y(λ∞M→(λ∞L<→(m|H≥
(≠Y.t≠88m
;Y+∧;Yλ∞M→;H∞M→(≠L↑h≤}.>→;(∞⎇⎇;→∧Y#"N<=λ~-d≠;⎇
≥{H_O∀_8⎇
≡X=~-lh≠{LT≠yH
≡≤h_l]≠≤kD
~→(∧
Y=h∧∞}<⎇](λ≥m};→β!.~→;D
;⎇Y$=x>$\[{$
=≤h
]⎇~→.$_;Y∧∞~→(∞∞[xy.>h≥{n]→λ≤nL<]λ
}Y<KD∧∃~→!Q\≥<N
|y(
|H≥~T→→<m≤{H≥l≡h≥≠d∧→→;-⎇\⎇≤L≡→(λ∞M_=λ∧<XZ.NX<Z-O(λ_m⎇<≠→/↓"X{mlZ9⎇.,=~;mnhλ_m};→λ∧Y(λ∞<;→K.,<≤[lN8z;Ltλ(∧∞~→(∧{{<
L>~=∂∀_Y:-lc"X.>⎇<Y,D_↑(≥≤{h∞,<=:.-;Yh∞M_=λ∞M→>(,(≥;M≡Y<\l≥λ_{m↑≥=→..kC"AQB4z-ly(≥M⎇H∪Y.]8;[D}h≥~-\+λ≤m≥<≠→.$_X<m≤h_y-M≤h_,M:=≥
≥Yc"N<;→K.,<≤[lN8z;Lt≥;Z.l<\x-D_{{.∞=→<N4~_=LT_Y9-d→~<l=⎇Y<L\C"JM→(≤m≥<≠→.>λ≤{dX<H
≡h≥~T≥≥{d∞⎇_=T∪~9LT_==
⎇8=≠md≠yH	-z≠Hλ={]x/∀
.&gj+C!*~→(∞>_=→$
yH_$y;≠∧=λ≥
≥9(¬NEl(~.4→→=↑[:;L\λ~=∞4≤⎇_.L(_=∧∞~;9$α]λβ!,;Yλ∞M→(≤nL=→<d
yH~.Nh→:,⎇≥λ≠L]9z_M}\h_.D≥~;,T¬]D∧∪X;,]≡+β!,(≤≠m≥]λ≥m
|y(∞>_=→$
<h∧∞z;≠∧z_;L|(≥≠d∞⎇_=T(~,d→>_,>≠≡(∞M≤Y9!Q[yH
≡≤h≠L]9z_M}\h_.,(~;D∞⎇_=T+H∧λ(≤≠m≥]λ≥m
|y(∞>_=→$
<h$∞z;≠↓Q\Y;,≥;H~-d≤⎇_.L((
≤H≥≥mt≠|H∞M≤Y9$
yH~.Nh≠Y-≤z_[n.h_<LT~;H∞>_=→$ε+C"I≥H_;
D≠⎇~↑H_x.<<h≥
(≤⎇≡→(_L\{{9.4≠|H∞,;8:-nhAQC"B(={]x/∀|h~-m=~8-D~;]]]λ≥l≡h≥≠d
;y→-D_(_M≡]~λ≥Yλ→\=~β!.≤[xl↑|h≥m<Y8O∀_(_l]≠λ~.4_[|Md
→{l↑h~;NMh≤⎇≡→(%∀~9H
≡λ~_.4≥~→!Qving neighborpεQ]C[∃YrAi!dβ↔∃Jβπ;⊃ε#'↔MεK→β' h+'Mε+'S#/⊃βS?zβ3?;.ce↓#FMβ;}s∃β?∩β?;∃εs↔'∨F∪?IβNqβOS∂#∃↓EJβ?IβO_4+?6+K∂K␈;∪↔⊃αC#πMε3?WIε{Iβ7␈∪∃β3O3';≥εs↔'∨F∪?KMJq↓α#*βπ3OzβπO/. 4+←F+S#↔∩β';≠Ns'S↔gIβ∨K␈;';≥ε≠?;≠N;WKπ&K?;Mπ;↔K∃πβ?OON∪3∃1ε;⊂4T;?OC/⊃β≠'.7"π∞-w6.D∞FF∂D∞FF/,Tπ>/,U`hU>↑'π⊗≡=⊗v>O∃BεOD∞G/⊗l\Bε␈↑Dπ&F≡Dπ≡.LeW⊗/∞-v'.=≥f:π]m↔6/.<⊗`h,=voπ↑LW↔~=w.fD&*ε.]⊗g"∞↑αε∂4	FN6T6}vm≤w/⊗≡M⊗}w5aPPh!_6}w=≤F/∩∀εw.\,W∩ε|dπ∨.=↓PW≡]Lbo⊗↑∞&}'\=⊗v:∞]fO6↑.6∞b=voπ↑LW↔~
}ε/⊗≡M⊗v:
≥bπ&T∧fNlQPWεL≥f*b≥f"π>↑πε␈<Tπ&F≡Dπ&F←∀εF∂lTε⊗.]dππ⊗|}&∞n\\Bπ&t∞7'.O⊃PW&Tππ⊗}W↔&≤↑2ε}d∞FF.≡$π>␈-LBε∞lDπ&z=vno]m⊗≡∂LTε∞n⎇lrπ&]W≡.NlW_h,≤&␈/D
↔"b∞W⊗F≡∞2ππ↑.7.Nltπ6∂-≥w/~⎇v∞g46zn}W⊗∂M≡f.g∀⊗v"=voε↑M↔&Ol]GJpQ(6∞fD∞FF/<Tε≡}lm⊗?/,≡FN}n4π⊗}-}G~pQ)⊗rπ=⎇V*π,↑7ε.>N2π&]↔∩ε≥nF.fL\7'.≥Dε∞vD∞6≡N]nFN6≤4ππ⊗|-F.o1Q'>NMDε⊗*
M⊗↑*
}W↔~D'/"
≥bε}lTεn∞-}"π⊗↑>ε.∨D∞FF/∀
FO6QQ&Nr∀π≡N↑
F/∩∞⎇w⊗fD∞FF∞d
w/↔4∞6..↑4π&z,Rrα	l⊗n.O∃Bh.Mε*εn]f&∞\]g&∞D∞εGO=≤7~ε|dπ&F]≡"π>}-F"ε≡4π&F≡Dε}2∞Mε*εM≤f(h,≡W&}\≡F}rD⊗v"∞Mε/⊗T
↔~εmtε}↔>L⊗≡fT∞Fzε\≤6Bπ-|&␈"αm6v␈⎇≥f:Q.FFO4∞εGO=≤7~b≥f"ε,]⊗v:≤&f*∞Mrπ≡≥↑Vf∂LQPW&Tε/6⎇NW&N⎇dε}2∀εfNlTε≡}lm⊗?/,≡FN}dvO6]dπ&FT
⊗vOM≤⊗`h.>F∂&Udα∧n},V␈6↑%BεNd∞FF*
≥fO&≤≥Bπ∨L≡F*ε|dπ&FT∞&}⊗}Dπ>␈-LBεO4fNv≡LRεOD6∞pQ-ε∂6T&..d∞&.≡},F."
≥bε.≤=απ⊗|-w"ε≥dπ&FT&.>≥mfNvt
w∩ε]N6(h.,V≡␈,LV"ε⎇dε
π>N&Oα
|bε≡]MG~πM↔"πMRπ⊗|-w'~<⊗rπ,\⊗"pQ%¬&FT
⊗v6≥m↔&*∞,V?⊗↑>2ε}d
ε∂6≥lrπ&tF/≡>-⊗⊗*∞Mε*εL↑6∨⊗≡∞FN}aQ&O~≡f}NL\Bε↔∀∩ε≡⎇nf.wM≥vrπM↔"πMRε&↑<7⊗O∞M⊗}r
≡2εv}APW≡↑↔⊗∂L]GJεL↑6∨⊗≤,V"b.W"ε<≥bε⊗T∞&.∞Dαf⊗␈M∧ε∂~∀ε&/<>&OπM≥vrε|dπ&FQQ'>␈-LB↓6≥lBε∂4∩ε&↑<7⊗O∞M⊗}r
|bεON<Vf2e⊃PPh!~6Nv<Tπ&F↑<Rπ⊗|-w'~
=f␈:∞Mε*ε≥m↔&N≥Dπ∨&≡LRε}d∞FF.≡$π>␈-L@hV≥lBεON4εf∂}4ε}2
]w&N⎇eBπ&←∩ε≡≥dπ≡N↑]F∂&T↔~ε↑\6@h-|bπ&]↔∩π⎇}&f"}4εFO>Mw↔J≡2π&←∩π>≥nBbε≡>7.n≥lrπ&≡Bε.≤=hV<≥bε?-}rεNnMrπ.m|6∨/
≤V"π>⊗≡*∞=rε∂4∞Fzε≡f*ε\]V␈↔∀∞Fzπ>Mw⊗(Q.FF*∞>F∂&↑4ε}2∞Mε*π⎇}&f",VNvt∞6No]L↔&.Edα¬&
≡2π≡≥↑Vf∂M≥vph-≡2εv\<W∨≡≡-⊗gJ
↑V≡B∞=F␈>↑$π&F≥dπ⊗.≥Dπ&N\UBπ≡t∞FF/∀6∞r
lW6/$6∂&=∧π/α∞⎇↔&B∞Mε(h.≤L↑y;]∧¬(≠→.D_;≠ml(≤≤L\~8⎇∧∞~→(n=≥<LUHλ∃

<h~.4_{→,≡H~9D∞y#"M≥89z-l(≥~T≤z;.]_=~-⎇H_x..Z99∧
⎇=λ
≥H_(∞>≤X:,⎇≥→[n.x<Y
O(_↑$∞<→_.M;Yc!,(≠~.>λ≠yD⎇<\L]]≠≡$8⎇~.l(_y-M≤h~-d≥~→$∞z;=-L=→9∧∞{|[A"X8l=|Y~-lh≥≠d∞~→(	M9Y(∞.;→+∧]=λ
≡λ_;∞=h_<∞
~9<d∞≠h_-o(_{↑Y<H
\=~→-\=~8l≥β"[,↑~≠y∧∞~_=∧
:9z∞D≤≤Y,M8⎇λ
];≠~-⎇\h≠ld≤⎇→.∞h_:\9H∧¬∀{{,T∪~9LQ"X{mlZ9⎇.,=~;mnkλ→%lkH≤nL=~8d
{Y<d
|H≠ml<h_m⎇]_:-m;Yh∞=;Y{T¬Y{
≤→<\d↓"[|DαXx;Mm{H_l≥H~_.l(≥~]<H→
≡⎇_;ND→]=∞↑Y<h∞∞Y9~,>→9λ∞⎇=~λ
M=≥≠Q"X{m↑≥=~-lkJ(∧	X;9-O+λ~,d≥~→. 2P;r\2P0wλ0v3w\4z46H37y⊂≤zqt⊂≤92r4Xz4wwεE0P≤7q7zλ1wzv→⊂12P≠pr2P≥40z⊂≥wzv2λ892r~qz⊂4]9P7{[⊂3:z≥y2P0[2⊂:4→wεE2~ywq2↑P:42H892r~qz4w[↔⊂⊂*~2P22]0tv2Y⊂897[s⊂;w]v2⊂1→P0w0[7swz\P:7FB:42P≤97wsλ7s⊂:[9wv;_q4v4]<P7sλ:42P~0v:4[3P89≠q62vH37y⊂∃:y4w→P6pqZ4w2yKεEεEαg7{P≥rP1w[rP:7H:42P≤7tw:λ7s⊂:~4yP6≠w3P2~yxzt\tz4w[↔⊂⊂)]x87yYFE;rH;tytλ:7P8≤7sy0[P0P9≠q7z⊂≥7P12H9zqqYyys:[⊂4w⊂≥42P&~s2P;[y62⊂~wεE1[vx2z~z4wwλ7y⊂1[Vwx2\0z4w[⊂;tz~⊂:42H7z42\9W⊂⊂∃tz47]z⊂0w≡P4r2XFE7sλ47{P≥7P3t]2P0P≠pz42[pz4qXv⊂89≠ws⊗⊂∩P;tv≠⊂1v0ZvP:4_z⊂7z\⊂97q≠zεE;Zv6⊂7→rr⊂8≤7sy0[yP:4_z⊂0yXy4q2H8:y8≠yryP_w2⊂1→v4rs≤P:7P~z9P3→v67{H97q7]9FE0[2⊂⊂8≤2r4q]⊂47{H:42|H;tv6λ92pq]⊂:7P~z9P⊂≠{wεE_qz4w[9P1<H0yyz[tw3P≥40z⊂	Y:42↑P;tv≠⊂0qzλ4w⊂;X|yP:~0z⊂:~2|P1→v4r{→FE;t[6⊂0qZ4r{2H:42t\⊂3wp[9RXWλ⊂'zyλ97q7]⊂6tsZ:⊂0q\zty2H:42yYP6rw≥0v⊂:~2wy4YyFE4[⊂9r{→y0v⊂≥p|y]λ⊂#4y≤z⊗⊂;YP6tsZ:⊂22\tsw⊂≥42P:[4{2y≤pv⊂6Xqt4w→P9wP≥40zεB:42|H0y2P≤92yr[:⊂4wλ:42P~w4z4Xv⊂1w[34sz\0z4w[⊂7s⊂≥42P;[y62↔λ⊂)rq[w2⊗⊂≥rP6tYt:εE≤97sy_vP4zλ:7P0Xxzty→P:44\P4r2XyP1<H4w2:Xz4wwλ397vH4z8β experience
and even transmit them to others through an "educational system".  Third,
it might derive the psychological laws from the fundamental physics of
the world and its knowledge of the initial configuration,
and finally, it might discover how robots are built from Life cells by doing
experimental "biology".

	Knowing the Life  physics without some information  about the
initial configuration is insufficient to derive the %2psychological%1
laws, because  robots can  be constructed  in the  Life  world in  an
infinity of ways.  This follows from the "folk theorem" that the Life
automaton  is universal in the sense  that any cellular automaton can
be constructed by taking sufficiently large squares of  Life cells as
the basic cell of the other automaton.$

	Men are in a more difficult intellectual position than
Life robots.  We don't know the fundamental physics of our world,
and we can't even be sure that its fundamental physics is describable
in finite terms.  Even if we knew the physical laws, they seem to
preclude precise knowledge of an initial state and precise calculation
of its future both for quantum mechanical reasons and because the
continuous functions needed to represent fields seem to involve
an infinite amount of information.

~ Our own ability to derive the laws of higher levels of organization
from knowledge  of lower level laws is  also limited by universality.
While the presentl accepted laws of physics allow only one chemistry,
the  laws of physics and chemistry allow many
biologies, and,  because the neuron is a universal computing element,
an arbitrary  mental structure is  allowed by basic  neurophysiology.
Therefore,  to  determine  human  mental  structure,  one  must  make
psychological  experiments,  ⊗or  determine  the  actual   anatomical
structure of the  brain and the  information stored in it .
One cannot determine the structure of the brain merely
from  the fact that the  brain is capable of  certain problem solving
performance.  In this  respect, our position
is similar to that of the Life robot.~

	One point of the cellular automaton robot  example is to make
plausible  the idea  that much  of human mental  structure is  not an
accident of evolution  or even of  the physics of  our world, but  is
r↓Kck%eKHA→←dAgUGGKgMMkXAAe←EY∃ZAg←1mS]N↓EKQCYS←dA¬]HA[UghAE∀AIKg%O]KH4∃S]i<A←dA∃m←Ym∃HAEr↓C]rAMsgiK4AiQCPAKqQ%ESif↓gkGP↓EKQCYS←d\4∀]gW%`@d~(~∀∩F8@Jgπ=[aki∃dAiS5J[gQ¬eS]N↓gsgi∃[f\JD@A)Q∃gJAG=[aYS
CiKH4∃G←[AkiKd↓ae←OIC[fA¬YY←G¬iJAG=[aki∃dAiS5JAC]⊂A←iQ∃dAeKM←keG∃fAC[=]NAkMKef\4∃)QKdACYY=nAKC
PAkg∃dA←L↓iQJA
←[akQKdAi<AEKQ¬mJACLAiQ←UOPAQ∀AQCH4∃BAG=[aki∃dA←L↓QSfA=o\XA	khAC1g↑AC1Y←nAQQKZAQ↑AgQ¬eJAM%YKfA=L~∃I¬iBAC9HAae=OeC[LAC]H↓i↑AG=[[k]%GCiJ↓oSiP↓KCGP↓←iQKH\~∃)!KrACIJA←MQK\AkMKHAM=dA[C9rAsK¬efAo%iPAG=]iS]UCXAg5CYXA
QC]O∃fXAC9H~∃C9HAiQ∀AaK←AYJA[¬WS]N↓iQJA
QC]O∃fAC]⊂AG←eIKGiS9NAKeI←ef~)CeJA=MiK\↓ISMM∃eK]h↓]←hAQQJA←ISOS]¬XACkQQ←ef↓←LAi!JAgsMiKZ\4∃αAa∃eg←\↓G←]MI←]iK⊂AoSi AiQJ↓iCgV↓←LAG=eeKGQS]NA∧A[CYon or
making a change in a time-sharing system can conveniently use a
mentalistic model of the system.

	Thus suppose a user complains that the system
will not run his program.  Perhaps the system believes that he
doesn't want to run, perhaps it persistently believes that he
has just run, perhaps it believes that his quota of computer
resources is exhausted, or perhaps it believes that his program
requires a resource that is unavailable.  Testing these hypotheses
can often be done with surprisingly little understanding of the
internal workings of the program.
.skip 2
	#. %3Programs designed to reason.%1  Suppose we explicitly design a
program to represent information by sentences in a certain language
stored in the memory of the computer and decide what to do by making
inferences, and doing what it concludes will advance its goals.  Naturally,
we would hope that our previous second order definition of belief will
"approve of" a %2B(p,s)%1 that ascribed to the program believing the
sentences explicitly built in.  We would be somewhat embarassed if
someone were to show that our second order definition approved as
well or better of an entirely different set of beliefs.

	Such a program was first proposed in (McCarthy 1960), and here is how
it might work:

	Information about the world is stored in a wide variety of
data structures.  For example, a visual scene received by a TV
camera may be represented by a 512x512x3 array of numbers representing
the intensities of three colors at the points of the visual field.
At another level, the same scene may be represented by a list of regions,
and at a further level there may be a list of physical objects and their
parts together with other information about these objects obtained from
non-visual sources.  Moreover, information about how to solve various
kinds of problems may be represented by programs in some programming
language.

	However, all the above representations are subordinate to
a collection of sentences in a suitable first order language that
includes set theory.  By subordinate, we mean that there are sentences
that tell what the data structures represent and what the programs do.
New sentences can arise by a variety of processes: inference from
sentences already present, by computation from the data structures representing
observations, ...

→→→→→There will be more here about what mental qualities should be programmed.←←←

.SKIP TO COLUMN 1
.ONCE CENTER
.bb |"GLOSSARY" OF MENTAL QUALITIES|

	In this section we give short "definitions" for machines of a collection
of mental qualities.  We include a number of terms which give us
difficulty with an indication of what the difficulties seem to be.

.ITEM←0;

	#. %3Actions%1.  We want to distinguish the actions of a being
from events that occur in its body and that affect the outside
world.  For example, we wish to distinguish a random twitch
from a purposeful movement.  This is not difficult %2relative
to a theory of belief that includes intentions%1.  One's purposeful
actions are those that would have been different had one's intentions
been different.  This requires that the theory of belief have sufficient
Cartesian product structure so that the counterfactual conditional
`"if its intentions had been different" is defined in the theory.
As explained in the section on definitions relative to an approximate
theory, it is not necessary that the counterfactual be given a
meaning in terms of the real world.

	#. %3Introspection and self-knowledge.%1

	We  say that  a  machine introspects  when it  comes  to have
beliefs about its own mental  state.  A simple form of  introspection
takes  place  when  a  program  determines  whether  it  has  certain
information  and if not asks for it.   Often an operating system will
compute a check  sum of itself  every few minutes  to verify that  it
hasn't been changed by a software or hardware malfunction.

	In principle,  introspection is easier  for computer programs
than for people, because the entire memory in which programs and data
are stored is available for inspection.   In fact, a computer program
can  be  made to  predict how  it  would react  to  particular inputs
provided it has enough free storage to perform the calculation.  This
situation smells of paradox, and there  is one.  Namely, if a program
could predict its own actions in less time than it takes to carry out
the action, it could refuse  to do what it has predicted  for itself.
This only  shows that self-simulation  is necessarily a  slow process,
and this is not surprising.

	However,   present    programs    do    little    interesting
introspection.   This is just  a matter of  the undeveloped  state of
artificial  intelligence; programmers  don't yet know  how to  make a
computer program look at itself in a useful way.

#.  %3Consciousness and self-consciousness%1.  Suppose we wish to
distinguish the self-awareness of a machine, animal or person from its
awareness of other things.  We explicate awareness as belief in certain
sentences, so in this case we are want to distinguish those sentences or
those terms in the sentences that may be considered to be about the self.
We also don't expect that self-consciousness will be a single property
that something either has or hasn't but rather there will be many kinds of
self-awareness with humans posessing many of the kinds we can imagine.

	Here are some of the kinds of self-awareness:
.subitem←0

	&. Certain predicates of the situation (propositional fluents
in the terminology of (McCarthy and Hayes 1970)) are directly observable
in almost all situations while others often must be inferred.
The almost always observable fluents may reasonably be identified
with the senses.  Likewise the values of certain fluents are almost
always under the control of the being and can be called motor
parameters for lack of a common language term.  We have in mind the
positions of the joints.
Most motor parameters are both observable and controllable.
I am inclined to regard the posession of a substantial set of
such constantly observable or controllable fluents as the most
primitive form of self-consciousness, but I have no strong arguments
against someone who wished to require more.

	&. The second level of self-consciousness requires a term
⊗I in the language denoting the self.  ⊗I should belong to the class
of persistent objects and some of the same predicates should be
applicable to it as are applicable to other objects.  For example,
like other objects ⊗I has a location that can change in time.  ⊗I is
also visible and impenetrable like other objects.  However, we don't
want to get carried away in regarding a physical body as a necessary
condition for self-consciousness.  Imagine a distributed computer
whose sense and motor organs could also be in a variety of places.
We don't want to exclude it from self-consciousness by definition.

	&. The third level come when ⊗I is regarded as an actor
among others.  The conditions that permit ⊗I to do something are
similar to the conditions that permit other actors to do similar
things.

	&. The fourth level requires the applicability of predicates
such as ⊗believes, ⊗wants and ⊗can to ⊗I.  Beliefs about past situations
and the ability to hypothesize future situations are also required
for this level.


#. %3Language and thought%1.  Here is a hypothesis arising from artificial
intelligence concerning the relation between language and thought.
Imagine a person or machine that represents information internally in a
huge network.  Each node of the network has references to other nodes
through relations.  (If the system has a variable collection of relations,
then the relations have to be represented by nodes, and we get a
symmetrical theory if we suppose that each node is connected to a set of
pairs of other nodes).  We can imagine this structure to have a long term
part and also extremely temporary parts representing current %2thoughts%1.
Naturally, each being has a its own network depending on its own
experience. A thought is then a temporary node currently being referenced
by the mechanism of consciousness.  Its meaning is determined by its
references to other nodes which in turn refer to yet other nodes.  Now
consider the problem of communicating a thought to another being.

	Its full communication would  involve transmitting the entire
network  that  can  be  reached  from  the  given  node, and this would
ordinarily constitute the entire experience of the being.   More than
that,  it would be  necessary to  also communicate the  programs that
that take action on the basis of encountering certain nodes.  Even if
all this could be transmitted, the recipient would still have to find
equivalents  for  the  information  in  terms  of  its  own  network.
Therefore, thoughts  have to  be translated  into  a public  language
before they can be commuunicated.

	A language  is also a network  of associations
and  programs.  However, certain  of the nodes  in this network (more
accurately a %2family%1 of networks, since no two people speak precisely the same
language) are  associated with words  or set phrases.  Sometimes the
translation from thoughts to sentences is easy,
because large  parts of the  private
networks are taken from the public network, and there is an advantage
in preserving the correspondence.  However, the translation is always
approximate (in sense that still lacks a technical definition),
and some  areas of
experience  are  difficult to  translate  at  all.  Sometimes this is for
intrinsic  reasons, and  sometimes because  particular cultures don't
use language in this area.  (It is my impression that cultures differ
in the  extent to which information about  facial appearance that can
be used for recognition is  verbally transmitted).  According to  this
scheme, the "deep structure"  of a publicly expressible thought  is a
node in the public network.  It is translated into the deep structure
of a sentence as a tree  whose terminal nodes are the nodes to  which
words or set  phrases are attached.  This "deep  structure" then must
be translated into a string in a spoken or written language.

	The need to use language to express thought also applies when
we have to ascribe thoughts to other beings, since  we cannot put the
entire network into a single sentence.

	#. %3Intentions.%1

	We may say that a machine intends to perform an action when it
believes that it will perform the action and it believes that the
action will further a goal.
However, further analysis may show that no such first order definition
in terms of belief adequately describes intentions.  In this case,
we can try a second order definition based on an axiomatization of
a predicate %2I(a,s)%1 meaning that the machine intends the action
⊗a when it is in state ⊗s.

	#. %3Free will%1

	When we program a computer  to  make
choices  intelligently  after determining its options,
examining their consequences, and deciding which
is  most  favorable or most moral or whatever, we must
program it  to  take  an  attitude  towards  its  freedom  of  choice
essentially  isomorphic  to  that  which a human must take to his own.

	We can define whether a particular action
was free or forced relative to a theory
that ascribes beliefs and within which
beings do what they believe will advance their goals.
In such a theory, action is precipitated by a belief of the form
%2I should do X now%1.  We will say that the action was free if
changing the belief to %2I shouldn't do X now%1 would have resulted
in the action not being performed.
This requires that the theory of belief have sufficient Cartesian
product structure so that changing a single belief is defined, but it
doesn't require defining what the state of the world would be if
a single belief were different.

	This isn't the  whole free will story, because  moralists are
also  concerned with whether praise  or blame may be  attributed to a
choice.   The following  considerations would  seem to  apply to  any
attempt to define  the morality of actions in a  way that would apply
to machines:

&. There is unlikely  to be a simple behavioral  definition.  Instead
there would be  a second order definition criticizing predicates that
ascribe morality to actions.


&. The theory must contain at least one axiom of morality that is not
just a statement of physical fact.  Relative to this axiom, moral
judgments of actions can be factual.

&. The theory of morality will presuppose a theory of belief in which
statements of the form %2"It believed the action would harm someone"%1
are defined.  The theory must ascribe beliefs about others' welfare and
perhaps about the being's own welfare.

&. It might be necessary to consider the  machine as imbedded in some
kind of society in order to ascribe morality to its actions.

&. No  present machines admit  such a belief  structure, and  no such
structure  may be required  to make  a machine with  arbitrarily high
intelligence in the sense of problem-solving ability.

&. It seems unlikely that morally  judgable machines or  machines to
which rights might legitimately be ascribed should be made if and when
it becomes possible to do so.

→→→→→→More mental qualities will be discussed.←←←←←←←←←

.SKIP TO COLUMN 1
.ONCE CENTER
.bb OTHER VIEWS ABOUT MIND

→→→→→This section will be written←←←←←
.SKIP TO COLUMN 1
.PORTION NOTES
.bb NOTES

.RECEIVE;

.ITEM←NOTE;
	#. Philosophy and artificial intelligence.  These fields overlap
in the following way:  In order to make a computer program behave
intelligently, its designer must build into it a view of the world
in general, apart from what they include about
particular sciences.  (The skeptic who doubts
whether there is anything to say about the world apart from the
particular sciences should try to write a computer program that can
figure out how to get to Timbuktoo, taking into account not only
the facts about travel in general but also facts about what people
and documents  have what information, and what information will
be required at different stages of the trip and when and how it is
to be obtained.  He will rapidly discover that he is lacking a %2science
of common sense%1, i.e. he will be unable to formally express and
build into his program "what everybody knows".  Maybe philosophy
could be defined as an attempted %2science of common sense%1,
or else the %2science of common sense%1 should be a definite part
of philosophy.)

	Artificial intelligence has a another component in which
philosophers have not studied, namely %2heuristics%1.  Heuristics
is  concerned  with:  given  the  facts  and  a  goal,  how should it
investigate the possibilities and decide what to do.
On the other hand, artificial intelligence is not much concerned
with aesthetics and ethics.

	Not all approaches to philosophy lead to results relevant to
the artificial intelligence problem.  On the face of it, a philosophy
that entailed the view that artificial intelligence was impossible
would be unhelpful, but besides that, taking artificial intelligence
seriously suggests some philosophical points of view.  I am not sure
that all I shall list are required for pursuing the AI goal -
some of them may be just my prejudices - but here they are:

		&. The relation between a world view and the world
should be studied by methods akin to metamathematics in which
systems are studied from the outside.  In metamathematics we study
the relation between a mathematical system and its models.  Philosophy
(or perhaps %2metaphilosophy%1) should study the relation between
world structures and systems within them that seek knowledge.
Just as the metamathematician can use any mathematical methods
in this study and distinguishes the methods he uses form those
being studied, so the philosopher should use all his scientific
knowledge in studying philosphical systems from the outside.

	Thus the question %2"How do I know?"%1 is best answered by studying
%2"How does it know"%1, getting the best answer that the current state
of science and philosophy permits, and then seeing how this answer stands
up to doubts about one's own sources of knowledge.

	&. We  regard  %2metaphysics%1 as  the study  of the  general
structure  of   the  world  and  %2epistemology%1  as  studying  what
knowledge of  the world  can be  had by  an  intelligence with  given
opportunities  to observe  and experiment.   We  need to  distinguish
what can  be determined  about the  structure of  humans and
machines  by  scientific   research  over  a   period  of  time   and
experimenting with many individuals from  what can be learned by in a
particular situation with particular opportunities to observe.   From
the AI point of view, the latter is as important
as the  former, and we suppose that  philosophers would also consider
it part of epistemology.  The possibilities of reductionism  are also
different  for  theoretical  and  everyday epistemology.    We  could
imagine that the rules of everyday epistemology could be deduced from
a knowledge of physics and the structure of the  being and the world,
but  we  can't see  how  one could  avoid  using  mental concepts  in
expressing knowledge actually obtained by the senses.

	&.  It  is  now  accepted that the basic concepts of physical
theories are far removed from observation.  The  human  sense  organs
are  many  levels  of  organization  removed  from quantum mechanical
states, and we have learned to accept the complication this causes in
verifying physical theories. Experience in trying to make intelligent
computer programs suggests that the  basic  concepts  of  the  common
sense  world  are  also complex and not always directly accessible to
observation.   In  particular,  the  common  sense  world  is  not  a
construct  from  sense  data,  but sense data play an important role.
When a man or a computer program sees a dog, we will  need  both  the
relation  between  the  observer and the dog and the relation between
the observer and the brown patch in order to construct a good  theory
of the event.

	&.  In spirit this paper is  materialist, but it is logically
compatible  with some  other philosophies.   Thus  cellular automaton
models of the  physical world may be  supplemented by supposing  that
certain  complex  configurations  interact with  additional  automata
called   souls   that  also   interact   with  each   other.     Such
%2interactionist  dualism%1   won't  meet   emotional  or   spiritual
objections to  materialism, but it does provide a logical  niche for any
empirically  argued  belief in  telepathy, communication with the dead
and  other   psychic phenomena.
A person who believed the alleged evidence for such phenomena and
still wanted a scientific explanation could model his beliefs
with auxiliary automata.

.SKIP TO COLUMN 1
.bb REFERENCES

%3Carnap, Rudolf%1 (1956), %2Meaning and Necessity%1, University of Chicago
Press.

%3McCarthy, J. and Hayes, P.J.%1 (1960∩$A'←[∀A!QS1←g←a!SGCX↓!e←E1K[fA→e←Z~)iQJAMiC]IA←S]h↓←LAβIiSMS
SCXA%]iKY1SOK]
J\@JI≠CGQ%]JA∪9iKYY%OK]G∀@hJb0~∃a`8@hlf4j`d@!KIfA5KYiu∃dXA∧8AC]H↓≠SGQ%JXAλ8R\A⊃S]EkIOPtA∃IS]EUeOP~)+]Sm∃egSidA!eKMf\~∀4∀222d22235←eJAIKMKe∃]GKf↓oSYX↓EJAgUaaYS∃I???⎇???>4∀]¬≥∪≤A-∃%¬β)%~~∀~)∃←Q\↓≠GπCIiQr~)βeiS→SGSC0A∪]i∃YYSO∃]GJA1CE←e¬i←er4∃'iC9M←eH↓+]Sm∃egSid~∃'i¬]M←e⊂XAπC1SM←e9SB@rPf`j~(]≥λ4∀